Forums & Discussions Home

    A place for members, communities, and committees to have discussions online and via e-mail.
    Click a category or topic to below to start the conversation...

    You are currently in view only mode for this forum. Please click the appropriate below to login as a member and participate. If you are not a member, please CLICK HERE for more information.


    Jan 23
    2013

    Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre LeBlanc
    [CBSA]
    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here's what I found:

    - Our application needs to process data by a given Timestamp, assigned based on priority.

    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes

    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table

    - All related data must be processed in the sequence it was captured.

    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.

    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.

    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately

    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first

    - Multiple threads compete to process data in this table

    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)


    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - ...

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit


    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock - waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock - waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don't see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I'm not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    André LeBlanc

    David Simpson
    [Themis Inc.]
    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.



    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email] <mailto:[login to unmask email]>

    For more information about Themis, visit www.themisinc.com http://www.themisinc.com



    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    Esteemed Listers, I can really use your help.



    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.

    And then it climbed to 8000 deadlocks per day!



    Here’s what I found:

    - Our application needs to process data by a given Timestamp, assigned based on priority.

    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes

    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table

    - All related data must be processed in the sequence it was captured.

    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.

    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.

    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately

    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first

    - Multiple threads compete to process data in this table

    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)



    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - …

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit



    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock – waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock



    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.



    We see 800 deadlocks a day when running 4 competing threads.

    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.



    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.

    There is retry logic, so the end user is not that concerned about the deadlocks.



    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.



    Cheers!

    Andr鍊 LeBlanc





    -----End Original Message-----

    Attachments
    • image001.jpg (2k)
    Andre LeBlanc
    [CBSA]
    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    [cid:[login to unmask email]______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:

    - Our application needs to process data by a given Timestamp, assigned based on priority.

    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes

    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table

    - All related data must be processed in the sequence it was captured.

    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.

    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.

    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately

    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first

    - Multiple threads compete to process data in this table

    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)


    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - …

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit


    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock – waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr鍊 LeBlanc


    -----End Original Message-----

    -----End Original Message-----
    Attachments
    • image001.jpg (2k)
    David Simpson
    [Themis Inc.]
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…



    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    I thought of that, but since it can get the S lock, would it skip the row in the fetch?



    Thanks,

    André LeBlanc



    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.



    cid:[login to unmask email]______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email] <mailto:[login to unmask email]>

    For more information about Themis, visit www.themisinc.com http://www.themisinc.com



    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    Esteemed Listers, I can really use your help.



    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.

    And then it climbed to 8000 deadlocks per day!



    Here’s what I found:

    - Our application needs to process data by a given Timestamp, assigned based on priority.

    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes

    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table

    - All related data must be processed in the sequence it was captured.

    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.

    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.

    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately

    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first

    - Multiple threads compete to process data in this table

    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)



    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - …

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit



    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock – waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock



    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.



    We see 800 deadlocks a day when running 4 competing threads.

    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.



    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.

    There is retry logic, so the end user is not that concerned about the deadlocks.



    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.



    Cheers!

    Andr鍊 LeBlanc





    -----End Original Message-----



    -----End Original Message-----



    -----End Original Message-----

    Charles Brown
    [BMI]
    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]> wrote:

    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]
    For more information about Themis, visit www.themisinc.com


    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock – waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr鍊 LeBlanc



    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    Site Links: View post online View mailing list online Send new post via email Unsubscribe from this mailing list Manage your subscription
    ** ** ** Attend the 2013 IDUG DB2 Tech Conference North America ** ** **
    ---> Caribe Royale All-Suites Hotel, Orlando, Florida, 29 April – 3 May, 2013 <---
    http://www.idug.org/p/cm/ld/fid=213

    Use of this email content is governed by the terms of service at:
    http://www.idug.org/p/cm/ld/fid=2
    Andre LeBlanc
    [CBSA]
    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    [cid:[login to unmask email]______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:

    - Our application needs to process data by a given Timestamp, assigned based on priority.

    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes

    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table

    - All related data must be processed in the sequence it was captured.

    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.

    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.

    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately

    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first

    - Multiple threads compete to process data in this table

    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)


    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - …

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit


    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock – waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr鍊 LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    Basivi Inaganti
    [Nationwide ]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________




    From: "LeBlanc,André-ITB" <[login to unmask email]>
    To: "'[login to unmask email]'" <[login to unmask email]>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    [DB2 z/OS v9]



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static
    db2 packages -- please advise because I couldn't find these pertinent info
    in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row
    in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA
    clause on your cursor select in DB2 9. This feature was intended for a
    situation like yours. If the cursor encounters a locked row it will skip
    it without waiting. I would certainly test this heavily in your
    environment to see if it’s appropriate.

    ______________________________________________________________________________

    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    [login to unmask email]
    For more information about Themis, visit www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught
    my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp,
    assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be
    processed within 2 minutes
    - Some rows may be related to other rows (A few or a few
    thousand) in a separate relationship table
    - All related data must be processed in the sequence it was
    captured.
    - Tablespace has 120 parts, partitioned and clustered by
    day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are
    somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based
    on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a
    separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will
    try to claim row
    - Update row using values from Fetch44 (will get +100 if row
    already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same
    transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock �C waits on Thread2 S
    lock
    4- Thread2 issues UPDATE and tries to get U lock �C waits on Thread1 U
    lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually
    don’t see any row locks for most Fetches. It appears that row locks may
    appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000
    deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer
    Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the
    deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have
    are greatly appreciated.

    Cheers!
    Andr�� LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----


    Site Links: View post online View mailing list online Send new post
    via email Unsubscribe from this mailing list Manage your subscription
    ** ** ** Attend the 2013 IDUG DB2 Tech Conference North America ** ** **
    ---> Caribe Royale All-Suites Hotel, Orlando, Florida, 29 April �C 3 May,
    2013 <---
    http://www.idug.org/p/cm/ld/fid=213

    Use of this email content is governed by the terms of service at:
    http://www.idug.org/p/cm/ld/fid=2


    Attachments
    • blankfilename.txt (7k)
    Walter Janißen
    [ITERGO]
    I am pretty sure that you can use the SKIP LOCKED DATA also for Update-Statements. So the second UPDATE would get +100 and knows, that the row is currently being process by another application.



    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996



    ________________________________
    Von: David Simpson [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 15:28
    An: [login to unmask email]
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    [cid:[login to unmask email]______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:

    - Our application needs to process data by a given Timestamp, assigned based on priority.

    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes

    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table

    - All related data must be processed in the sequence it was captured.

    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.

    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.

    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately

    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first

    - Multiple threads compete to process data in this table

    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)


    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - …

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit


    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock – waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr鍊 LeBlanc


    -----End Original Message-----

    -----End Original Message-----
    Attachments
    • image001.jpg (2k)
    Walter Janißen
    [ITERGO]
    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996



    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]>
    To: "'[login to unmask email]'" <[login to unmask email]>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It's only supposed to skip incompatible locks (I think)... Test Test Test...

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I'm not sure it's a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it's appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here's what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - ...
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don't see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I'm not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    Attachments
    • ATT00001.gif (7k)
    Andre LeBlanc
    [CBSA]
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It's only supposed to skip incompatible locks (I think)... Test Test Test...

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I'm not sure it's a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it's appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here's what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - ...
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don't see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I'm not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----
    Joe Geller
    [J Geller Software]

    Andre,

    Just to make sure - you say the update does not move the row to another partition, but does it update the partitioning columns?  And, is this update statement the only processing you do on the row (i.e., the cusor is not defined as FOR UPDATE)?

    Joe

    In Reply to Andre LeBlanc:

    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It's only supposed to skip incompatible locks (I think)... Test Test Test...

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I'm not sure it's a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it's appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here's what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - ...
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don't see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I'm not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----
    Andre LeBlanc
    [CBSA]
    I hadn’t thought of using SKIP LOCKED DATA on the UPDATE, but I think it’s worth looking into to resolve the deadlocks.

    I’m also interested in hearing your thoughts on how we defined the CURSOR and PARTITIONS to sequence the data.
    Should we use SCROLLABLE, DYNAMIC, FOR UPDATE OF, or ORDER BY on this Cursor? The current cursor does not update or use any rowset options.
    Is this an effective use of Partitions, or would a SEGMENTED TS be better?
    Finally, since we are in Data Sharing, should we have ROW level locks or PAGE locks with MAXROWS = 1?

    Thanks!
    Andre


    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:26 AM
    To: [login to unmask email]
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I am pretty sure that you can use the SKIP LOCKED DATA also for Update-Statements. So the second UPDATE would get +100 and knows, that the row is currently being process by another application.



    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: David Simpson [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 15:28
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    [cid:[login to unmask email]______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:

    - Our application needs to process data by a given Timestamp, assigned based on priority.

    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes

    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table

    - All related data must be processed in the sequence it was captured.

    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.

    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.

    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately

    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first

    - Multiple threads compete to process data in this table

    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)


    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - …

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit


    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock – waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr鍊 LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    Charles Brown
    [BMI]
    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]> wrote:

    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.
    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996



    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________




    From: "LeBlanc,André-ITB" <[login to unmask email]>
    To: "'[login to unmask email]'" <[login to unmask email]>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]
    For more information about Themis, visit www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----


    Site Links: View post online View mailing list online Send new post via email Unsubscribe from this mailing list Manage your subscription
    ** ** ** Attend the 2013 IDUG DB2 Tech Conference North America ** ** **
    ---> Caribe Royale All-Suites Hotel, Orlando, Florida, 29 April ?C 3 May, 2013 <---
    http://www.idug.org/p/cm/ld/fid=213

    Use of this email content is governed by the terms of service at:
    http://www.idug.org/p/cm/ld/fid=2


    -----End Original Message-----

    -----End Original Message-----

    Site Links: View post online View mailing list online Send new post via email Unsubscribe from this mailing list Manage your subscription
    ** ** ** Attend the 2013 IDUG DB2 Tech Conference North America ** ** **
    ---> Caribe Royale All-Suites Hotel, Orlando, Florida, 29 April – 3 May, 2013 <---
    http://www.idug.org/p/cm/ld/fid=213

    Use of this email content is governed by the terms of service at:
    http://www.idug.org/p/cm/ld/fid=2
    Wayne Campbell
    [Administrative Office of the Courts]
    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.
    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]<mailto:[login to unmask email]>'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----
    Andre LeBlanc
    [CBSA]
    The high number of deadlocks also concerned me, which is why I’m trying to come up with a better solution for what they’re trying to accomplish.
    Unfortunately, they are their own worst enemy since the deadlocks all come from the same application logic running in parallel on multiple threads.
    There are no uncommitted inserts or adhoc queries involved.

    I was hoping to get them to commit after the select, but their application logic only allows them to commit after the update.

    But I am curious about high number of deadlocks:

    - How many deadlocks are too many?

    - Can excessive deadlocks create problems for DB2? (Other than giving me headaches…)

    Thanks!
    Andre

    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: January 23, 2013 4:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.
    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]<mailto:[login to unmask email]>'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----
    Daniel Luksetich
    [DanL Database Consulting]
    The first question to ask is the work getting done? If the answer is yes,
    then the urgency of the matter is not critical, and so you can take a
    meaningful and balanced approach to a resolution.

    The excessive deadlocks are probably resulting in resource consumption in
    access of what a well behaving application would consume. So, if you are
    getting the work done then you are just running a little "hot". Remember,
    you don't get deadlocks for free, you have to code them into your
    application, and so the solution will probably be application based. When I
    design high volume processing systems, especially batch systems as you have
    described, I employ what I call "application controlled parallelism". This
    is where multiple streams are launched in parallel to process the data in
    "logical" partitions. In this way we can avoid all sorts of contention and
    achieve processing rates in excess of 10,000 per second. In contrast, you
    have what I call "application out of control parallelism". I have seen this
    many times before where people use CICS as a way to launch parallel batch
    streams. The same thing happens all the time with locking and other
    contention.

    The first thing I would examine is whether or not you can single thread
    the process. If so, you can see how one thread performs and gauge the
    impact of the contention. Second, I would see if you can some how control
    the parallelism by getting each thread to work on something different. This
    is probably more difficult to do in CICS then batch. You may need to refine
    your feeder process to somehow split streams to feed to CICS. If you can
    accomplish something here then that may all that needs to be done. If you
    can't then I like what some people have suggested. However, instead of WITH
    UR on the read I would use FOR UPDATE OF to lock the row immediately, but
    do this in combination with SKIP LOCKED DATA or you will be single
    threading again. Test this out and see if it reduces contention. it may be
    the easiest thing to do, and result in an "application almost in control
    parallelism".

    I hope you find this helpful.

    Dan

    On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    <[login to unmask email]> wrote:
    > The high number of deadlocks also concerned me, which is why I’m trying
    to
    > come up with a better solution for what they’re trying to accomplish.
    > Unfortunately, they are their own worst enemy since the deadlocks all
    come
    > from the same application logic running in parallel on multiple threads.
    > There are no uncommitted inserts or adhoc queries involved.
    >
    > I was hoping to get them to commit after the select, but their
    application
    > logic only allows them to commit after the update.
    >
    > But I am curious about high number of deadlocks:
    >
    > - How many deadlocks are too many?
    >
    > - Can excessive deadlocks create problems for DB2? (Other than
    > giving me headaches…)
    >
    > Thanks!
    > Andre
    >
    > From: Campbell, Wayne [mailto:[login to unmask email]
    > Sent: January 23, 2013 4:42 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Remember to use “WITH UR” only if it’s appropriate for your needs. Are
    > you seeing any lock escalations? What is your commit logic like? Do
    you
    > commit after select statement? This got us a little while back, the
    > developers didn’t realize that selects took locks and left them hanging.
    > Wayne Campbell
    > DB2 DBA
    > Administrative Office of the Courts
    > (360) 705-5268
    > Email: [login to unmask email]<mailto:[login to unmask email]>
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 10:33 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > More than 500 within a day -- never heard of that many deadlocks not
    even
    > within a month period. Is there something fundamentally wrong -- you
    bet!!
    > For a starter, i would suggest you update all your Ad hoc READs to use
    "
    > ...WITH UR".
    > Next increase your zparm timeout parm Take advantage of RCT parm
    > ROLBE=NO. And finally take advantage of some of the lock avoidance
    parms
    > available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    your
    > bind parms look good.
    >
    > If you're still getting more than 20 a day -- I'll suggest its time you
    > create anther SUBSYS. Some applications just can't coexist.
    >
    > Thx and best,
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > wrote:
    > In this case, the update does not move the row to another partition.
    >
    > From: Walter Janißen [mailto:[login to unmask email]
    > Sent: January 23, 2013 10:27 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > But that is no longer a problem as in some earlier DB2 versions.
    >
    > Mit freundlichen Grüßen
    > Walter Janißen
    >
    > ITERGO Informationstechnologie GmbH
    > Anwendungsentwicklung
    > Technische Anwendungsarchitektur
    > Victoriaplatz 2
    > D-40198 Düsseldorf
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > ITERGO Informationstechnologie GmbH
    > Vorsitzender des Aufsichtsrats: Christian Diedrich
    > Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    > Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    Regauer
    > Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >
    >
    > ________________________________
    > Von: Basivi Inaganti [mailto:[login to unmask email]
    > Gesendet: Mittwoch, 23. Januar 2013 16:19
    > An: [login to unmask email]<mailto:[login to unmask email]>
    > Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    > Andre,
    >
    > By any chance, is the update causing the rows to move to new partition?
    >
    > Thanks,
    > ______________________________________________________________________
    > [cid:[login to unmask email]
    >
    >
    >
    > From: "LeBlanc,André-ITB"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > Date: 01/23/2013 10:01 AM
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    my
    > [DB2 z/OS v9]
    > ________________________________
    >
    >
    >
    > You are correct Charles, these are CICS COBOL packages.
    > Here are the bind parameters:
    >
    > RELEASE ( COMMIT ) ISOLATION ( CS )
    > VALIDATE ( BIND ) EXPLAIN ( YES )
    > DYNAMICRULES ( )
    > DEGREE ( 1 ) CURRENTDATA ( NO )
    > KEEPDYNAMIC ( NO )
    > DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    > PATHDEFAULT
    > IMMEDWRITE ( NO ) ENCODING ( 37 )
    >
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:48 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > Andre
    > What level of db2 concurrency are you using? My guess, these are static
    > db2 packages -- please advise because I couldn't find these pertinent
    info
    > in your problem description.
    > Best and Thx !!!
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 8:30 AM, David Simpson
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > It’s only supposed to skip incompatible locks (I think)… Test Test Test…
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:30 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > I thought of that, but since it can get the S lock, would it skip the
    row
    > in the fetch?
    >
    > Thanks,
    > André LeBlanc
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:28 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    DATA
    > clause on your cursor select in DB2 9. This feature was intended for a
    > situation like yours. If the cursor encounters a locked row it will
    skip
    > it without waiting. I would certainly test this heavily in your
    > environment to see if it’s appropriate.
    >
    >
    ______________________________________________________________________________
    > David Simpson | Senior Technical Advisor | Themis Education
    > 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    > [login to unmask email]<mailto:[login to unmask email]>
    > For more information about Themis, visit
    > www.themisinc.comhttp://www.themisinc.com
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:23 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it caught
    > my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks may
    > appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to 8000
    > deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    > -----End Original Message-----
    >
    > -----End Original Message-----
    Dee Reins
    [Centurylink]
    We have data sharing, and had problems with locking. We carefully added row level locking to the tables we needed to, and that resolved our issue. Row level locking should not be a problem in a single DB2 environment. I suggest picking a table with the most locks, make it row level locking. The next table will bubble to the top, repeat the process. Doing this with “WITH UR”, should help a lot.





    "Leadership and learning are indispensable to each other." - John Fitzgerald Kennedy

    360 905-7343 Jan 31 new number is 360 905-7943
    360 448-8623 (c)

    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 1:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.
    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]<mailto:[login to unmask email]>'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----
    David Simpson
    [Themis Inc.]
    Let me (re)-understand the problem. Each concurrent task retrieves a row to begin processing and attempts to lock it down by issuing an update statement. If this is true WITH UR may only shift the problem to the update and cause excessive waits rather than timeouts. Using WITH UR both competing tasks will still attempt to do the same piece of work. I believe SKIP LOCKED DATA offers the best chance for success here since once TASK A issues its update, TASK B will not attempt to read the row.



    However, if the situation is more complex than my simple understanding, of course there could be other issues.



    From: Reins, Dee D [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 5:06 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    We have data sharing, and had problems with locking. We carefully added row level locking to the tables we needed to, and that resolved our issue. Row level locking should not be a problem in a single DB2 environment. I suggest picking a table with the most locks, make it row level locking. The next table will bubble to the top, repeat the process. Doing this with “WITH UR”, should help a lot.











    "Leadership and learning are indispensable to each other." - John Fitzgerald Kennedy



    360 905-7343 Jan 31 new number is 360 905-7943

    360 448-8623 (c)



    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 1:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.

    Wayne Campbell

    DB2 DBA

    Administrative Office of the Courts

    (360) 705-5268

    Email: [login to unmask email]



    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    Andre

    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.



    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.



    Thx and best,


    Sent from my iPhone


    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]> wrote:

    In this case, the update does not move the row to another partition.



    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email] <mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996





    ________________________________

    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________




    From: "LeBlanc,André-ITB" <[login to unmask email]>
    To: "'[login to unmask email]'" <[login to unmask email]>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    ________________________________




    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email] <mailto:[login to unmask email]> > wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email] <mailto:[login to unmask email]> '
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email] <mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email] <mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.com http://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email] <mailto:[login to unmask email]> '
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    David Simpson
    [Themis Inc.]
    Sorry… third sentence should read “cause excessive waits and timeouts instead of deadlocks…”





    From: David Simpson [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 6:05 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    Let me (re)-understand the problem. Each concurrent task retrieves a row to begin processing and attempts to lock it down by issuing an update statement. If this is true WITH UR may only shift the problem to the update and cause excessive waits rather than timeouts. Using WITH UR both competing tasks will still attempt to do the same piece of work. I believe SKIP LOCKED DATA offers the best chance for success here since once TASK A issues its update, TASK B will not attempt to read the row.



    However, if the situation is more complex than my simple understanding, of course there could be other issues.



    From: Reins, Dee D [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 5:06 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    We have data sharing, and had problems with locking. We carefully added row level locking to the tables we needed to, and that resolved our issue. Row level locking should not be a problem in a single DB2 environment. I suggest picking a table with the most locks, make it row level locking. The next table will bubble to the top, repeat the process. Doing this with “WITH UR”, should help a lot.











    "Leadership and learning are indispensable to each other." - John Fitzgerald Kennedy



    360 905-7343 Jan 31 new number is 360 905-7943

    360 448-8623 (c)



    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 1:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.

    Wayne Campbell

    DB2 DBA

    Administrative Office of the Courts

    (360) 705-5268

    Email: [login to unmask email]



    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    Andre

    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.



    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.



    Thx and best,


    Sent from my iPhone


    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]> wrote:

    In this case, the update does not move the row to another partition.



    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]



    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email] <mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996





    ________________________________

    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________




    From: "LeBlanc,André-ITB" <[login to unmask email]>
    To: "'[login to unmask email]'" <[login to unmask email]>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    ________________________________




    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email] <mailto:[login to unmask email]> > wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email] <mailto:[login to unmask email]> '
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email] <mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email] <mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.com http://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email] <mailto:[login to unmask email]> ]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email] <mailto:[login to unmask email]> '
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    Wayne Campbell
    [Administrative Office of the Courts]
    You never mentioned if your experiencing any lock escalations to the table. If you are just increasing the NUMLKTS might solve your problems.

    To anser your questions,

    In a perfect world ,any deadlocks are too many. But since we don’t live in a perfect world, DB2 is built to handle deadlocks and we can handle them, but there is a cost involved, more CPU longer elapse time, ect. Is 800 deadlocks to many? If your updating the table 100 million times ,maybe not. If you are updating it a thousand time, probably.

    For the second questions YES! You gone from 800 to 8,000 deadlocks in a timeframe you didn’t mention But I assume it wasn’t that long So in the same amount of time you can probably expect it to go to 80,000. At some point performance will start to degrade significantly. That what I’ve seen in the past. I would start getting the word out to anyone that will listen or not, developer, your supervisor, management. Do it in emails so it’s well documented. CYA

    Wouldn’t be nice to work in a perfect would with no deadlocks, timeouts, and developers would commit after select? The thing I have about committing properly? That I got from Bonnie Baker somewhere along the line and have used it to solve contention and performance problems in the past. Basically anything Bonnie says is gospel to me.


    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 1:51 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    The high number of deadlocks also concerned me, which is why I’m trying to come up with a better solution for what they’re trying to accomplish.
    Unfortunately, they are their own worst enemy since the deadlocks all come from the same application logic running in parallel on multiple threads.
    There are no uncommitted inserts or adhoc queries involved.

    I was hoping to get them to commit after the select, but their application logic only allows them to commit after the update.

    But I am curious about high number of deadlocks:

    - How many deadlocks are too many?

    - Can excessive deadlocks create problems for DB2? (Other than giving me headaches…)

    Thanks!
    Andre
    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: January 23, 2013 4:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.
    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]<mailto:[login to unmask email]>'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----
    Ted MacNeil
    Are your SLA's being met?

    Is response trending upwards with the deadlocks?

    Yes to 1. NP.
    Yes to 2. Further work.

    No to both. NEXT!

    But, monitor.
    -
    Ted MacNEIL
    [login to unmask email]
    Twitter: @TedMacNEIL

    -----Original Message-----
    From: "Campbell, Wayne" <[login to unmask email]>
    Date: Wed, 23 Jan 2013 16:33:59
    To: '[login to unmask email]'<[login to unmask email]>
    Reply-To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2
    z/OS v9]

    You never mentioned if your experiencing any lock escalations to the table. If you are just increasing the NUMLKTS might solve your problems.

    To anser your questions,

    In a perfect world ,any deadlocks are too many. But since we don’t live in a perfect world, DB2 is built to handle deadlocks and we can handle them, but there is a cost involved, more CPU longer elapse time, ect. Is 800 deadlocks to many? If your updating the table 100 million times ,maybe not. If you are updating it a thousand time, probably.

    For the second questions YES! You gone from 800 to 8,000 deadlocks in a timeframe you didn’t mention But I assume it wasn’t that long So in the same amount of time you can probably expect it to go to 80,000. At some point performance will start to degrade significantly. That what I’ve seen in the past. I would start getting the word out to anyone that will listen or not, developer, your supervisor, management. Do it in emails so it’s well documented. CYA

    Wouldn’t be nice to work in a perfect would with no deadlocks, timeouts, and developers would commit after select? The thing I have about committing properly? That I got from Bonnie Baker somewhere along the line and have used it to solve contention and performance problems in the past. Basically anything Bonnie says is gospel to me.


    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 1:51 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    The high number of deadlocks also concerned me, which is why I’m trying to come up with a better solution for what they’re trying to accomplish.
    Unfortunately, they are their own worst enemy since the deadlocks all come from the same application logic running in parallel on multiple threads.
    There are no uncommitted inserts or adhoc queries involved.

    I was hoping to get them to commit after the select, but their application logic only allows them to commit after the update.

    But I am curious about high number of deadlocks:

    - How many deadlocks are too many?

    - Can excessive deadlocks create problems for DB2? (Other than giving me headaches…)

    Thanks!
    Andre
    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: January 23, 2013 4:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.
    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]<mailto:[login to unmask email]>'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----

    -----End Original Message-----

    Ted MacNeil
    Whoops!
    My bad!

    Yes & no. NEXT!

    Sorry!


    -
    Ted MacNEIL
    [login to unmask email]
    Twitter: @TedMacNEIL

    -----Original Message-----
    From: Ted MacNeil <[login to unmask email]>
    Date: Thu, 24 Jan 2013 03:44:56
    To: DB2-L<[login to unmask email]>
    Reply-To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2
    z/OS v9]

    Are your SLA's being met?

    Is response trending upwards with the deadlocks?

    Yes to 1. NP.
    Yes to 2. Further work.

    No to both. NEXT!

    But, monitor.
    -
    Ted MacNEIL
    [login to unmask email]
    Twitter: @TedMacNEIL

    -----Original Message-----
    From: "Campbell, Wayne" <[login to unmask email]>
    Date: Wed, 23 Jan 2013 16:33:59
    To: '[login to unmask email]'<[login to unmask email]>
    Reply-To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2
    z/OS v9]

    You never mentioned if your experiencing any lock escalations to the table. If you are just increasing the NUMLKTS might solve your problems.

    To anser your questions,

    In a perfect world ,any deadlocks are too many. But since we don’t live in a perfect world, DB2 is built to handle deadlocks and we can handle them, but there is a cost involved, more CPU longer elapse time, ect. Is 800 deadlocks to many? If your updating the table 100 million times ,maybe not. If you are updating it a thousand time, probably.

    For the second questions YES! You gone from 800 to 8,000 deadlocks in a timeframe you didn’t mention But I assume it wasn’t that long So in the same amount of time you can probably expect it to go to 80,000. At some point performance will start to degrade significantly. That what I’ve seen in the past. I would start getting the word out to anyone that will listen or not, developer, your supervisor, management. Do it in emails so it’s well documented. CYA

    Wouldn’t be nice to work in a perfect would with no deadlocks, timeouts, and developers would commit after select? The thing I have about committing properly? That I got from Bonnie Baker somewhere along the line and have used it to solve contention and performance problems in the past. Basically anything Bonnie says is gospel to me.


    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 1:51 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    The high number of deadlocks also concerned me, which is why I’m trying to come up with a better solution for what they’re trying to accomplish.
    Unfortunately, they are their own worst enemy since the deadlocks all come from the same application logic running in parallel on multiple threads.
    There are no uncommitted inserts or adhoc queries involved.

    I was hoping to get them to commit after the select, but their application logic only allows them to commit after the update.

    But I am curious about high number of deadlocks:

    - How many deadlocks are too many?

    - Can excessive deadlocks create problems for DB2? (Other than giving me headaches…)

    Thanks!
    Andre
    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: January 23, 2013 4:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.
    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]<mailto:[login to unmask email]>'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----

    -----End Original Message-----


    -----End Original Message-----

    Andre LeBlanc
    [CBSA]
    You've hit the nail on the head Dan, this is precisely "application out of control parallelism".
    They needed higher throughput so they increased from 1 to 4 threads doing the same things and went to 800 deadlocks/day overnight.
    They then increased to 6 threads and doubled the throughput at the cost of 8000 deadlocks/day.
    As soon as they went back to 4 threads, back to 800.

    Given that the deadlock victims have no data changes to rollback, I don't think the occasional deadlocks cost as much as the constant inefficiency.
    I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get to the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    The 5 victims then continue fetching until the next UPDATE "FREE FOR ALL" claims 4 more victims... and so on

    Hopefully they'll reconsider this parallelism strategy.

    In the meantime, I'm also concerned that the ordering of the rows is based on our partitioning scheme instead of an explicit ORDER BY.
    The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions for the order?

    I really appreciate all the feedback on this.
    Thanks!
    Andre

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 23, 2013 5:07 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    The first question to ask is the work getting done? If the answer is yes,
    then the urgency of the matter is not critical, and so you can take a
    meaningful and balanced approach to a resolution.

    The excessive deadlocks are probably resulting in resource consumption in
    access of what a well behaving application would consume. So, if you are
    getting the work done then you are just running a little "hot". Remember,
    you don't get deadlocks for free, you have to code them into your
    application, and so the solution will probably be application based. When I
    design high volume processing systems, especially batch systems as you have
    described, I employ what I call "application controlled parallelism". This
    is where multiple streams are launched in parallel to process the data in
    "logical" partitions. In this way we can avoid all sorts of contention and
    achieve processing rates in excess of 10,000 per second. In contrast, you
    have what I call "application out of control parallelism". I have seen this
    many times before where people use CICS as a way to launch parallel batch
    streams. The same thing happens all the time with locking and other
    contention.

    The first thing I would examine is whether or not you can single thread
    the process. If so, you can see how one thread performs and gauge the
    impact of the contention. Second, I would see if you can some how control
    the parallelism by getting each thread to work on something different. This
    is probably more difficult to do in CICS then batch. You may need to refine
    your feeder process to somehow split streams to feed to CICS. If you can
    accomplish something here then that may all that needs to be done. If you
    can't then I like what some people have suggested. However, instead of WITH
    UR on the read I would use FOR UPDATE OF to lock the row immediately, but
    do this in combination with SKIP LOCKED DATA or you will be single
    threading again. Test this out and see if it reduces contention. it may be
    the easiest thing to do, and result in an "application almost in control
    parallelism".

    I hope you find this helpful.

    Dan

    On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    <[login to unmask email]> wrote:
    > The high number of deadlocks also concerned me, which is why I’m trying
    to
    > come up with a better solution for what they’re trying to accomplish.
    > Unfortunately, they are their own worst enemy since the deadlocks all
    come
    > from the same application logic running in parallel on multiple threads.
    > There are no uncommitted inserts or adhoc queries involved.
    >
    > I was hoping to get them to commit after the select, but their
    application
    > logic only allows them to commit after the update.
    >
    > But I am curious about high number of deadlocks:
    >
    > - How many deadlocks are too many?
    >
    > - Can excessive deadlocks create problems for DB2? (Other than
    > giving me headaches…)
    >
    > Thanks!
    > Andre
    >
    > From: Campbell, Wayne [mailto:[login to unmask email]
    > Sent: January 23, 2013 4:42 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Remember to use “WITH UR” only if it’s appropriate for your needs. Are
    > you seeing any lock escalations? What is your commit logic like? Do
    you
    > commit after select statement? This got us a little while back, the
    > developers didn’t realize that selects took locks and left them hanging.
    > Wayne Campbell
    > DB2 DBA
    > Administrative Office of the Courts
    > (360) 705-5268
    > Email: [login to unmask email]<mailto:[login to unmask email]>
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 10:33 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > More than 500 within a day -- never heard of that many deadlocks not
    even
    > within a month period. Is there something fundamentally wrong -- you
    bet!!
    > For a starter, i would suggest you update all your Ad hoc READs to use
    "
    > ...WITH UR".
    > Next increase your zparm timeout parm Take advantage of RCT parm
    > ROLBE=NO. And finally take advantage of some of the lock avoidance
    parms
    > available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    your
    > bind parms look good.
    >
    > If you're still getting more than 20 a day -- I'll suggest its time you
    > create anther SUBSYS. Some applications just can't coexist.
    >
    > Thx and best,
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > wrote:
    > In this case, the update does not move the row to another partition.
    >
    > From: Walter Janißen [mailto:[login to unmask email]
    > Sent: January 23, 2013 10:27 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > But that is no longer a problem as in some earlier DB2 versions.
    >
    > Mit freundlichen Grüßen
    > Walter Janißen
    >
    > ITERGO Informationstechnologie GmbH
    > Anwendungsentwicklung
    > Technische Anwendungsarchitektur
    > Victoriaplatz 2
    > D-40198 Düsseldorf
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > ITERGO Informationstechnologie GmbH
    > Vorsitzender des Aufsichtsrats: Christian Diedrich
    > Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    > Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    Regauer
    > Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >
    >
    > ________________________________
    > Von: Basivi Inaganti [mailto:[login to unmask email]
    > Gesendet: Mittwoch, 23. Januar 2013 16:19
    > An: [login to unmask email]<mailto:[login to unmask email]>
    > Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    > Andre,
    >
    > By any chance, is the update causing the rows to move to new partition?
    >
    > Thanks,
    > ______________________________________________________________________
    > [cid:[login to unmask email]
    >
    >
    >
    > From: "LeBlanc,André-ITB"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > Date: 01/23/2013 10:01 AM
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    my
    > [DB2 z/OS v9]
    > ________________________________
    >
    >
    >
    > You are correct Charles, these are CICS COBOL packages.
    > Here are the bind parameters:
    >
    > RELEASE ( COMMIT ) ISOLATION ( CS )
    > VALIDATE ( BIND ) EXPLAIN ( YES )
    > DYNAMICRULES ( )
    > DEGREE ( 1 ) CURRENTDATA ( NO )
    > KEEPDYNAMIC ( NO )
    > DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    > PATHDEFAULT
    > IMMEDWRITE ( NO ) ENCODING ( 37 )
    >
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:48 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > Andre
    > What level of db2 concurrency are you using? My guess, these are static
    > db2 packages -- please advise because I couldn't find these pertinent
    info
    > in your problem description.
    > Best and Thx !!!
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 8:30 AM, David Simpson
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > It’s only supposed to skip incompatible locks (I think)… Test Test Test…
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:30 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > I thought of that, but since it can get the S lock, would it skip the
    row
    > in the fetch?
    >
    > Thanks,
    > André LeBlanc
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:28 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    DATA
    > clause on your cursor select in DB2 9. This feature was intended for a
    > situation like yours. If the cursor encounters a locked row it will
    skip
    > it without waiting. I would certainly test this heavily in your
    > environment to see if it’s appropriate.
    >
    >
    ______________________________________________________________________________
    > David Simpson | Senior Technical Advisor | Themis Education
    > 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    > [login to unmask email]<mailto:[login to unmask email]>
    > For more information about Themis, visit
    > www.themisinc.comhttp://www.themisinc.com
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:23 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it caught
    > my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks may
    > appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to 8000
    > deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    > -----End Original Message-----
    >
    > -----End Original Message-----

    -----End Original Message-----

    Jay Reavill
    Interesting... So it sounds like the app just continues processing when a deadlock is hit. No outstanding updates to commit, so no real backout needs to be done. Just continues on its merry way. If that's the case, then in a crude way, its already doing SKIP LOCK DATA via deadlock. ;-) Obviously, not desirable, but would seem to be getting the same end result.

    Cheers,
    Jay


     
    Jay Reavill | Debit IT | Database Administrator
    Office: 727.227.2144 | Cell: 727.215.5794
    : [login to unmask email]



    -----Original Message-----
    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Thursday, January 24, 2013 10:03 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    You've hit the nail on the head Dan, this is precisely "application out of control parallelism".
    They needed higher throughput so they increased from 1 to 4 threads doing the same things and went to 800 deadlocks/day overnight.
    They then increased to 6 threads and doubled the throughput at the cost of 8000 deadlocks/day.
    As soon as they went back to 4 threads, back to 800.

    Given that the deadlock victims have no data changes to rollback, I don't think the occasional deadlocks cost as much as the constant inefficiency.
    I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get to the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    The 5 victims then continue fetching until the next UPDATE "FREE FOR ALL" claims 4 more victims... and so on

    Hopefully they'll reconsider this parallelism strategy.

    In the meantime, I'm also concerned that the ordering of the rows is based on our partitioning scheme instead of an explicit ORDER BY.
    The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions for the order?

    I really appreciate all the feedback on this.
    Thanks!
    Andre

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 23, 2013 5:07 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    The first question to ask is the work getting done? If the answer is yes, then the urgency of the matter is not critical, and so you can take a meaningful and balanced approach to a resolution.

    The excessive deadlocks are probably resulting in resource consumption in access of what a well behaving application would consume. So, if you are getting the work done then you are just running a little "hot". Remember, you don't get deadlocks for free, you have to code them into your application, and so the solution will probably be application based. When I design high volume processing systems, especially batch systems as you have described, I employ what I call "application controlled parallelism". This is where multiple streams are launched in parallel to process the data in "logical" partitions. In this way we can avoid all sorts of contention and achieve processing rates in excess of 10,000 per second. In contrast, you have what I call "application out of control parallelism". I have seen this many times before where people use CICS as a way to launch parallel batch streams. The same thing happens all the time with locking and other contention.

    The first thing I would examine is whether or not you can single thread the process. If so, you can see how one thread performs and gauge the impact of the contention. Second, I would see if you can some how control the parallelism by getting each thread to work on something different. This is probably more difficult to do in CICS then batch. You may need to refine your feeder process to somehow split streams to feed to CICS. If you can accomplish something here then that may all that needs to be done. If you can't then I like what some people have suggested. However, instead of WITH UR on the read I would use FOR UPDATE OF to lock the row immediately, but do this in combination with SKIP LOCKED DATA or you will be single threading again. Test this out and see if it reduces contention. it may be the easiest thing to do, and result in an "application almost in control parallelism".

    I hope you find this helpful.

    Dan

    On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB <[login to unmask email]> wrote:
    > The high number of deadlocks also concerned me, which is why I’m
    > trying
    to
    > come up with a better solution for what they’re trying to accomplish.
    > Unfortunately, they are their own worst enemy since the deadlocks all
    come
    > from the same application logic running in parallel on multiple threads.
    > There are no uncommitted inserts or adhoc queries involved.
    >
    > I was hoping to get them to commit after the select, but their
    application
    > logic only allows them to commit after the update.
    >
    > But I am curious about high number of deadlocks:
    >
    > - How many deadlocks are too many?
    >
    > - Can excessive deadlocks create problems for DB2? (Other than
    > giving me headaches…)
    >
    > Thanks!
    > Andre
    >
    > From: Campbell, Wayne [mailto:[login to unmask email]
    > Sent: January 23, 2013 4:42 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Remember to use “WITH UR” only if it’s appropriate for your needs.
    > Are you seeing any lock escalations? What is your commit logic like?
    > Do
    you
    > commit after select statement? This got us a little while back, the
    > developers didn’t realize that selects took locks and left them hanging.
    > Wayne Campbell
    > DB2 DBA
    > Administrative Office of the Courts
    > (360) 705-5268
    > Email:
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 10:33 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > More than 500 within a day -- never heard of that many deadlocks not
    even
    > within a month period. Is there something fundamentally wrong -- you
    bet!!
    > For a starter, i would suggest you update all your Ad hoc READs to
    > use
    "
    > ...WITH UR".
    > Next increase your zparm timeout parm Take advantage of RCT parm
    > ROLBE=NO. And finally take advantage of some of the lock avoidance
    parms
    > available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    your
    > bind parms look good.
    >
    > If you're still getting more than 20 a day -- I'll suggest its time
    > you create anther SUBSYS. Some applications just can't coexist.
    >
    > Thx and best,
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > wrote:
    > In this case, the update does not move the row to another partition.
    >
    > From: Walter Janißen [mailto:[login to unmask email]
    > Sent: January 23, 2013 10:27 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > But that is no longer a problem as in some earlier DB2 versions.
    >
    > Mit freundlichen Grüßen
    > Walter Janißen
    >
    > ITERGO Informationstechnologie GmbH
    > Anwendungsentwicklung
    > Technische Anwendungsarchitektur
    > Victoriaplatz 2
    > D-40198 Düsseldorf
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > ITERGO Informationstechnologie GmbH
    > Vorsitzender des Aufsichtsrats: Christian Diedrich
    > Geschäftsführung: Dr. Bettina Anders (Vorsitzende), Lothar Engelke,
    > Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    Regauer
    > Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >
    >
    > ________________________________
    > Von: Basivi Inaganti [mailto:[login to unmask email]
    > Gesendet: Mittwoch, 23. Januar 2013 16:19
    > An: [login to unmask email]<mailto:[login to unmask email]>
    > Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9] Andre,
    >
    > By any chance, is the update causing the rows to move to new partition?
    >
    > Thanks,
    > ______________________________________________________________________
    > [cid:[login to unmask email]
    >
    >
    >
    > From: "LeBlanc,André-ITB"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > Date: 01/23/2013 10:01 AM
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    my
    > [DB2 z/OS v9]
    > ________________________________
    >
    >
    >
    > You are correct Charles, these are CICS COBOL packages.
    > Here are the bind parameters:
    >
    > RELEASE ( COMMIT ) ISOLATION ( CS )
    > VALIDATE ( BIND ) EXPLAIN ( YES )
    > DYNAMICRULES ( )
    > DEGREE ( 1 ) CURRENTDATA ( NO )
    > KEEPDYNAMIC ( NO )
    > DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    > PATHDEFAULT
    > IMMEDWRITE ( NO ) ENCODING ( 37 )
    >
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:48 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > What level of db2 concurrency are you using? My guess, these are
    > static
    > db2 packages -- please advise because I couldn't find these pertinent
    info
    > in your problem description.
    > Best and Thx !!!
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 8:30 AM, David Simpson
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > It’s only supposed to skip incompatible locks (I think)… Test Test
    > Test…
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:30 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > I thought of that, but since it can get the S lock, would it skip the
    row
    > in the fetch?
    >
    > Thanks,
    > André LeBlanc
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:28 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    DATA
    > clause on your cursor select in DB2 9. This feature was intended for
    > a situation like yours. If the cursor encounters a locked row it will
    skip
    > it without waiting. I would certainly test this heavily in your
    > environment to see if it’s appropriate.
    >
    >
    ______________________________________________________________________________
    > David Simpson | Senior Technical Advisor | Themis Education
    > 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    > [login to unmask email]<mailto:[login to unmask email]>
    > For more information about Themis, visit
    > www.themisinc.comhttp://www.themisinc.com
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:23 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it
    > caught my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks
    > may appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to
    > 8000 deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    > -----End Original Message-----
    >
    > -----End Original Message-----

    -----End Original Message-----


    -----End Original Message-----


    _____________
    The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
    Daniel Luksetich
    [DanL Database Consulting]
    By the way, I'm not criticizing this method of parallelism, but
    emphasizing the fact that there are better ways to do this work. I have a
    customer that sells up to a million products per day. Doesn't seem like
    much in today's world, but they process/track each order, customer, and
    product during the entire life cycle of the sale, shipment, and payment.
    This results in many millions of transactions. Besides customer
    interaction, the entire process is online batch controlled via MQSeries,
    Batch, and CICS. It employs "almost in control parallelism" in that various
    critical components are launched in series. However, once a process is
    started there is no control of the parallelism and we get deadlocks. In
    fact, many thousands of deadlocks each day. It's a fact of a day's work at
    this place, and they accept it. Can they make it better? You bet. However,
    it's an application change and that would cost more money than what the
    deadlocks cost.

    If you are streaming data through this table, which is what it sounds to
    me like what you are doing, then clustering is irrelevant as the data will
    always be disorganized. If you can identify partitions in the driving query
    then you could assign threads to partitions via a control table. If you can
    identify a "type" of record to process then you could do the same thing
    with that column. This would completely eliminate your deadlocks. If none
    of that is possible you could try something crazy like adding a RAND()
    predicate to the query to make your parallelism "completely out of
    control". This could, in theory, reduce your deadlocks by inverse of the
    percentage in your predicate (e.g. AND RAND() < .2 is an 80% reduction).
    You won't eliminate deadlocks, and there might be a slight risk to meeting
    your SLA (but probably not greater than the risk associated with the
    deadlocks in this regard), but it is a one line change in the program and
    you can always increase the parallelism.

    Your current process is like soldiers storming a beach in a single file.
    As one goes down the next climbs over his back and continues the fight.
    Adding the RAND() predicate would scatter the soldiers. You'll still lose a
    few of them, but secure the beach faster and with less casualties.

    hmmm, this sounds like a conversation we should be having after work over
    a beer.

    Cheers,
    Dan

    P.S. I'm patenting the "application completely out of control parallelism"
    model so no one get any smart ideas, eh?

    On Thu, 24 Jan 2013 15:02:51 +0000, LeBlanc, André-ITB
    <[login to unmask email]> wrote:
    > You've hit the nail on the head Dan, this is precisely "application out
    of
    > control parallelism".
    > They needed higher throughput so they increased from 1 to 4 threads
    doing
    > the same things and went to 800 deadlocks/day overnight.
    > They then increased to 6 threads and doubled the throughput at the cost
    of
    > 8000 deadlocks/day.
    > As soon as they went back to 4 threads, back to 800.
    >
    > Given that the deadlock victims have no data changes to rollback, I
    don't
    > think the occasional deadlocks cost as much as the constant
    inefficiency.
    > I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get
    to
    > the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    > The 5 victims then continue fetching until the next UPDATE "FREE FOR
    ALL"
    > claims 4 more victims... and so on
    >
    > Hopefully they'll reconsider this parallelism strategy.
    >
    > In the meantime, I'm also concerned that the ordering of the rows is
    based
    > on our partitioning scheme instead of an explicit ORDER BY.
    > The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions
    > for the order?
    >
    > I really appreciate all the feedback on this.
    > Thanks!
    > Andre
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: January 23, 2013 5:07 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > The first question to ask is the work getting done? If the answer is
    yes,
    > then the urgency of the matter is not critical, and so you can take a
    > meaningful and balanced approach to a resolution.
    >
    > The excessive deadlocks are probably resulting in resource consumption
    in
    > access of what a well behaving application would consume. So, if you are
    > getting the work done then you are just running a little "hot".
    Remember,
    > you don't get deadlocks for free, you have to code them into your
    > application, and so the solution will probably be application based.
    When I
    > design high volume processing systems, especially batch systems as you
    have
    > described, I employ what I call "application controlled parallelism".
    This
    > is where multiple streams are launched in parallel to process the data
    in
    > "logical" partitions. In this way we can avoid all sorts of contention
    and
    > achieve processing rates in excess of 10,000 per second. In contrast,
    you
    > have what I call "application out of control parallelism". I have seen
    this
    > many times before where people use CICS as a way to launch parallel
    batch
    > streams. The same thing happens all the time with locking and other
    > contention.
    >
    > The first thing I would examine is whether or not you can single thread
    > the process. If so, you can see how one thread performs and gauge the
    > impact of the contention. Second, I would see if you can some how
    control
    > the parallelism by getting each thread to work on something different.
    This
    > is probably more difficult to do in CICS then batch. You may need to
    refine
    > your feeder process to somehow split streams to feed to CICS. If you can
    > accomplish something here then that may all that needs to be done. If
    you
    > can't then I like what some people have suggested. However, instead of
    WITH
    > UR on the read I would use FOR UPDATE OF to lock the row immediately,
    but
    > do this in combination with SKIP LOCKED DATA or you will be single
    > threading again. Test this out and see if it reduces contention. it may
    be
    > the easiest thing to do, and result in an "application almost in control
    > parallelism".
    >
    > I hope you find this helpful.
    >
    > Dan
    >
    > On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    > <[login to unmask email]> wrote:
    >> The high number of deadlocks also concerned me, which is why I’m trying
    > to
    >> come up with a better solution for what they’re trying to accomplish.
    >> Unfortunately, they are their own worst enemy since the deadlocks all
    > come
    >> from the same application logic running in parallel on multiple
    threads.
    >> There are no uncommitted inserts or adhoc queries involved.
    >>
    >> I was hoping to get them to commit after the select, but their
    > application
    >> logic only allows them to commit after the update.
    >>
    >> But I am curious about high number of deadlocks:
    >>
    >> - How many deadlocks are too many?
    >>
    >> - Can excessive deadlocks create problems for DB2? (Other
    than
    >> giving me headaches…)
    >>
    >> Thanks!
    >> Andre
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: January 23, 2013 4:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use “WITH UR” only if it’s appropriate for your needs. Are
    >> you seeing any lock escalations? What is your commit logic like? Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn’t realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    you
    >> create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    >> Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >> Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    >> It’s only supposed to skip incompatible locks (I think)… Test Test
    Test…
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for a
    >> situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it’s appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    caught
    >> my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here’s what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - …
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don’t see any row locks for most Fetches. It appears that row locks
    may
    >> appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    8000
    >> deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I’m not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----
    Andre LeBlanc
    [CBSA]
    Even though the data is disorganized, working on the data with the earliest Complete-by-Timestamp is critical.
    Some work has to be completed within minutes, while most work has hours to be processed.
    The concern is that we'll miss the tight deadlines without proper ordering.

    Thanks,
    André LeBlanc
    Performance Management - Data Management
    613-952-9570


    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 24, 2013 10:33 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    By the way, I'm not criticizing this method of parallelism, but
    emphasizing the fact that there are better ways to do this work. I have a
    customer that sells up to a million products per day. Doesn't seem like
    much in today's world, but they process/track each order, customer, and
    product during the entire life cycle of the sale, shipment, and payment.
    This results in many millions of transactions. Besides customer
    interaction, the entire process is online batch controlled via MQSeries,
    Batch, and CICS. It employs "almost in control parallelism" in that various
    critical components are launched in series. However, once a process is
    started there is no control of the parallelism and we get deadlocks. In
    fact, many thousands of deadlocks each day. It's a fact of a day's work at
    this place, and they accept it. Can they make it better? You bet. However,
    it's an application change and that would cost more money than what the
    deadlocks cost.

    If you are streaming data through this table, which is what it sounds to
    me like what you are doing, then clustering is irrelevant as the data will
    always be disorganized. If you can identify partitions in the driving query
    then you could assign threads to partitions via a control table. If you can
    identify a "type" of record to process then you could do the same thing
    with that column. This would completely eliminate your deadlocks. If none
    of that is possible you could try something crazy like adding a RAND()
    predicate to the query to make your parallelism "completely out of
    control". This could, in theory, reduce your deadlocks by inverse of the
    percentage in your predicate (e.g. AND RAND() < .2 is an 80% reduction).
    You won't eliminate deadlocks, and there might be a slight risk to meeting
    your SLA (but probably not greater than the risk associated with the
    deadlocks in this regard), but it is a one line change in the program and
    you can always increase the parallelism.

    Your current process is like soldiers storming a beach in a single file.
    As one goes down the next climbs over his back and continues the fight.
    Adding the RAND() predicate would scatter the soldiers. You'll still lose a
    few of them, but secure the beach faster and with less casualties.

    hmmm, this sounds like a conversation we should be having after work over
    a beer.

    Cheers,
    Dan

    P.S. I'm patenting the "application completely out of control parallelism"
    model so no one get any smart ideas, eh?

    On Thu, 24 Jan 2013 15:02:51 +0000, LeBlanc, André-ITB
    <[login to unmask email]> wrote:
    > You've hit the nail on the head Dan, this is precisely "application out
    of
    > control parallelism".
    > They needed higher throughput so they increased from 1 to 4 threads
    doing
    > the same things and went to 800 deadlocks/day overnight.
    > They then increased to 6 threads and doubled the throughput at the cost
    of
    > 8000 deadlocks/day.
    > As soon as they went back to 4 threads, back to 800.
    >
    > Given that the deadlock victims have no data changes to rollback, I
    don't
    > think the occasional deadlocks cost as much as the constant
    inefficiency.
    > I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get
    to
    > the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    > The 5 victims then continue fetching until the next UPDATE "FREE FOR
    ALL"
    > claims 4 more victims... and so on
    >
    > Hopefully they'll reconsider this parallelism strategy.
    >
    > In the meantime, I'm also concerned that the ordering of the rows is
    based
    > on our partitioning scheme instead of an explicit ORDER BY.
    > The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions
    > for the order?
    >
    > I really appreciate all the feedback on this.
    > Thanks!
    > Andre
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: January 23, 2013 5:07 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > The first question to ask is the work getting done? If the answer is
    yes,
    > then the urgency of the matter is not critical, and so you can take a
    > meaningful and balanced approach to a resolution.
    >
    > The excessive deadlocks are probably resulting in resource consumption
    in
    > access of what a well behaving application would consume. So, if you are
    > getting the work done then you are just running a little "hot".
    Remember,
    > you don't get deadlocks for free, you have to code them into your
    > application, and so the solution will probably be application based.
    When I
    > design high volume processing systems, especially batch systems as you
    have
    > described, I employ what I call "application controlled parallelism".
    This
    > is where multiple streams are launched in parallel to process the data
    in
    > "logical" partitions. In this way we can avoid all sorts of contention
    and
    > achieve processing rates in excess of 10,000 per second. In contrast,
    you
    > have what I call "application out of control parallelism". I have seen
    this
    > many times before where people use CICS as a way to launch parallel
    batch
    > streams. The same thing happens all the time with locking and other
    > contention.
    >
    > The first thing I would examine is whether or not you can single thread
    > the process. If so, you can see how one thread performs and gauge the
    > impact of the contention. Second, I would see if you can some how
    control
    > the parallelism by getting each thread to work on something different.
    This
    > is probably more difficult to do in CICS then batch. You may need to
    refine
    > your feeder process to somehow split streams to feed to CICS. If you can
    > accomplish something here then that may all that needs to be done. If
    you
    > can't then I like what some people have suggested. However, instead of
    WITH
    > UR on the read I would use FOR UPDATE OF to lock the row immediately,
    but
    > do this in combination with SKIP LOCKED DATA or you will be single
    > threading again. Test this out and see if it reduces contention. it may
    be
    > the easiest thing to do, and result in an "application almost in control
    > parallelism".
    >
    > I hope you find this helpful.
    >
    > Dan
    >
    > On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    > <[login to unmask email]> wrote:
    >> The high number of deadlocks also concerned me, which is why I’m trying
    > to
    >> come up with a better solution for what they’re trying to accomplish.
    >> Unfortunately, they are their own worst enemy since the deadlocks all
    > come
    >> from the same application logic running in parallel on multiple
    threads.
    >> There are no uncommitted inserts or adhoc queries involved.
    >>
    >> I was hoping to get them to commit after the select, but their
    > application
    >> logic only allows them to commit after the update.
    >>
    >> But I am curious about high number of deadlocks:
    >>
    >> - How many deadlocks are too many?
    >>
    >> - Can excessive deadlocks create problems for DB2? (Other
    than
    >> giving me headaches…)
    >>
    >> Thanks!
    >> Andre
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: January 23, 2013 4:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use “WITH UR” only if it’s appropriate for your needs. Are
    >> you seeing any lock escalations? What is your commit logic like? Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn’t realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    you
    >> create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    >> Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >> Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    >> It’s only supposed to skip incompatible locks (I think)… Test Test
    Test…
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for a
    >> situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it’s appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    caught
    >> my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here’s what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - …
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don’t see any row locks for most Fetches. It appears that row locks
    may
    >> appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    8000
    >> deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I’m not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----

    -----End Original Message-----

    Joe Geller
    [J Geller Software]

    In regards to Dan's suggestion of having different threads work on different rows; a system I'm supporting had a similar type of process, but with 40 or more threads.  They too were getting locking problems.  The solution used was to add a column with a thread number.  Each inserted row would get the next thread number (rotating from 1 to #threads).  The threads that process the rows (to do the update) would only read rows with the corresponding thread #) - completely avoiding the contention.

    The one tricky thing is to handle the ability to change the number of threads.  If you increase it, there is no problem. But if you decrease it, then there will be existing rows with a higher thread # than the number of threads, so you need to handle that (perhaps using something like MOD(thread#, #threads))

    Joe


    In Reply to Andre LeBlanc:

    Even though the data is disorganized, working on the data with the earliest Complete-by-Timestamp is critical.
    Some work has to be completed within minutes, while most work has hours to be processed.
    The concern is that we'll miss the tight deadlines without proper ordering.

    Thanks,
    André LeBlanc
    Performance Management - Data Management
    613-952-9570


    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 24, 2013 10:33 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    By the way, I'm not criticizing this method of parallelism, but
    emphasizing the fact that there are better ways to do this work. I have a
    customer that sells up to a million products per day. Doesn't seem like
    much in today's world, but they process/track each order, customer, and
    product during the entire life cycle of the sale, shipment, and payment.
    This results in many millions of transactions. Besides customer
    interaction, the entire process is online batch controlled via MQSeries,
    Batch, and CICS. It employs "almost in control parallelism" in that various
    critical components are launched in series. However, once a process is
    started there is no control of the parallelism and we get deadlocks. In
    fact, many thousands of deadlocks each day. It's a fact of a day's work at
    this place, and they accept it. Can they make it better? You bet. However,
    it's an application change and that would cost more money than what the
    deadlocks cost.

    If you are streaming data through this table, which is what it sounds to
    me like what you are doing, then clustering is irrelevant as the data will
    always be disorganized. If you can identify partitions in the driving query
    then you could assign threads to partitions via a control table. If you can
    identify a "type" of record to process then you could do the same thing
    with that column. This would completely eliminate your deadlocks. If none
    of that is possible you could try something crazy like adding a RAND()
    predicate to the query to make your parallelism "completely out of
    control". This could, in theory, reduce your deadlocks by inverse of the
    percentage in your predicate (e.g. AND RAND() < .2 is an 80% reduction).
    You won't eliminate deadlocks, and there might be a slight risk to meeting
    your SLA (but probably not greater than the risk associated with the
    deadlocks in this regard), but it is a one line change in the program and
    you can always increase the parallelism.

    Your current process is like soldiers storming a beach in a single file.
    As one goes down the next climbs over his back and continues the fight.
    Adding the RAND() predicate would scatter the soldiers. You'll still lose a
    few of them, but secure the beach faster and with less casualties.

    hmmm, this sounds like a conversation we should be having after work over
    a beer.

    Cheers,
    Dan

    P.S. I'm patenting the "application completely out of control parallelism"
    model so no one get any smart ideas, eh?

    On Thu, 24 Jan 2013 15:02:51 +0000, LeBlanc, André-ITB
    <[login to unmask email]> wrote:
    > You've hit the nail on the head Dan, this is precisely "application out
    of
    > control parallelism".
    > They needed higher throughput so they increased from 1 to 4 threads
    doing
    > the same things and went to 800 deadlocks/day overnight.
    > They then increased to 6 threads and doubled the throughput at the cost
    of
    > 8000 deadlocks/day.
    > As soon as they went back to 4 threads, back to 800.
    >
    > Given that the deadlock victims have no data changes to rollback, I
    don't
    > think the occasional deadlocks cost as much as the constant
    inefficiency.
    > I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get
    to
    > the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    > The 5 victims then continue fetching until the next UPDATE "FREE FOR
    ALL"
    > claims 4 more victims... and so on
    >
    > Hopefully they'll reconsider this parallelism strategy.
    >
    > In the meantime, I'm also concerned that the ordering of the rows is
    based
    > on our partitioning scheme instead of an explicit ORDER BY.
    > The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions
    > for the order?
    >
    > I really appreciate all the feedback on this.
    > Thanks!
    > Andre
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: January 23, 2013 5:07 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > The first question to ask is the work getting done? If the answer is
    yes,
    > then the urgency of the matter is not critical, and so you can take a
    > meaningful and balanced approach to a resolution.
    >
    > The excessive deadlocks are probably resulting in resource consumption
    in
    > access of what a well behaving application would consume. So, if you are
    > getting the work done then you are just running a little "hot".
    Remember,
    > you don't get deadlocks for free, you have to code them into your
    > application, and so the solution will probably be application based.
    When I
    > design high volume processing systems, especially batch systems as you
    have
    > described, I employ what I call "application controlled parallelism".
    This
    > is where multiple streams are launched in parallel to process the data
    in
    > "logical" partitions. In this way we can avoid all sorts of contention
    and
    > achieve processing rates in excess of 10,000 per second. In contrast,
    you
    > have what I call "application out of control parallelism". I have seen
    this
    > many times before where people use CICS as a way to launch parallel
    batch
    > streams. The same thing happens all the time with locking and other
    > contention.
    >
    > The first thing I would examine is whether or not you can single thread
    > the process. If so, you can see how one thread performs and gauge the
    > impact of the contention. Second, I would see if you can some how
    control
    > the parallelism by getting each thread to work on something different.
    This
    > is probably more difficult to do in CICS then batch. You may need to
    refine
    > your feeder process to somehow split streams to feed to CICS. If you can
    > accomplish something here then that may all that needs to be done. If
    you
    > can't then I like what some people have suggested. However, instead of
    WITH
    > UR on the read I would use FOR UPDATE OF to lock the row immediately,
    but
    > do this in combination with SKIP LOCKED DATA or you will be single
    > threading again. Test this out and see if it reduces contention. it may
    be
    > the easiest thing to do, and result in an "application almost in control
    > parallelism".
    >
    > I hope you find this helpful.
    >
    > Dan
    >
    > On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    > <[login to unmask email]> wrote:
    >> The high number of deadlocks also concerned me, which is why I’m trying
    > to
    >> come up with a better solution for what they’re trying to accomplish.
    >> Unfortunately, they are their own worst enemy since the deadlocks all
    > come
    >> from the same application logic running in parallel on multiple
    threads.
    >> There are no uncommitted inserts or adhoc queries involved.
    >>
    >> I was hoping to get them to commit after the select, but their
    > application
    >> logic only allows them to commit after the update.
    >>
    >> But I am curious about high number of deadlocks:
    >>
    >> - How many deadlocks are too many?
    >>
    >> - Can excessive deadlocks create problems for DB2? (Other
    than
    >> giving me headaches…)
    >>
    >> Thanks!
    >> Andre
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: January 23, 2013 4:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use “WITH UR” only if it’s appropriate for your needs. Are
    >> you seeing any lock escalations? What is your commit logic like? Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn’t realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    you
    >> create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    >> Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >> Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    >> It’s only supposed to skip incompatible locks (I think)… Test Test
    Test…
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for a
    >> situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it’s appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    caught
    >> my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here’s what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - …
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don’t see any row locks for most Fetches. It appears that row locks
    may
    >> appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    8000
    >> deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I’m not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----

    -----End Original Message-----

    Andre LeBlanc
    [CBSA]
    We'll be discussing something similar this afternoon... although there always seems to be some other business rule that complicates the whole process.

    I think I'll have that beer now... :)


    From: Joe Geller [mailto:[login to unmask email]
    Sent: January 24, 2013 11:05 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]


    In regards to Dan's suggestion of having different threads work on different rows; a system I'm supporting had a similar type of process, but with 40 or more threads. They too were getting locking problems. The solution used was to add a column with a thread number. Each inserted row would get the next thread number (rotating from 1 to #threads). The threads that process the rows (to do the update) would only read rows with the corresponding thread #) - completely avoiding the contention.

    The one tricky thing is to handle the ability to change the number of threads. If you increase it, there is no problem. But if you decrease it, then there will be existing rows with a higher thread # than the number of threads, so you need to handle that (perhaps using something like MOD(thread#, #threads))

    Joe

    In Reply to Andre LeBlanc:
    Even though the data is disorganized, working on the data with the earliest Complete-by-Timestamp is critical.
    Some work has to be completed within minutes, while most work has hours to be processed.
    The concern is that we'll miss the tight deadlines without proper ordering.

    Andre

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 24, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    By the way, I'm not criticizing this method of parallelism, but
    emphasizing the fact that there are better ways to do this work. I have a
    customer that sells up to a million products per day. Doesn't seem like
    much in today's world, but they process/track each order, customer, and
    product during the entire life cycle of the sale, shipment, and payment.
    This results in many millions of transactions. Besides customer
    interaction, the entire process is online batch controlled via MQSeries,
    Batch, and CICS. It employs "almost in control parallelism" in that various
    critical components are launched in series. However, once a process is
    started there is no control of the parallelism and we get deadlocks. In
    fact, many thousands of deadlocks each day. It's a fact of a day's work at
    this place, and they accept it. Can they make it better? You bet. However,
    it's an application change and that would cost more money than what the
    deadlocks cost.

    If you are streaming data through this table, which is what it sounds to
    me like what you are doing, then clustering is irrelevant as the data will
    always be disorganized. If you can identify partitions in the driving query
    then you could assign threads to partitions via a control table. If you can
    identify a "type" of record to process then you could do the same thing
    with that column. This would completely eliminate your deadlocks. If none
    of that is possible you could try something crazy like adding a RAND()
    predicate to the query to make your parallelism "completely out of
    control". This could, in theory, reduce your deadlocks by inverse of the
    percentage in your predicate (e.g. AND RAND() < .2 is an 80% reduction).
    You won't eliminate deadlocks, and there might be a slight risk to meeting
    your SLA (but probably not greater than the risk associated with the
    deadlocks in this regard), but it is a one line change in the program and
    you can always increase the parallelism.

    Your current process is like soldiers storming a beach in a single file.
    As one goes down the next climbs over his back and continues the fight.
    Adding the RAND() predicate would scatter the soldiers. You'll still lose a
    few of them, but secure the beach faster and with less casualties.

    hmmm, this sounds like a conversation we should be having after work over
    a beer.

    Cheers,
    Dan

    P.S. I'm patenting the "application completely out of control parallelism"
    model so no one get any smart ideas, eh?

    On Thu, 24 Jan 2013 15:02:51 +0000, LeBlanc, André-ITB
    <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > You've hit the nail on the head Dan, this is precisely "application out
    of
    > control parallelism".
    > They needed higher throughput so they increased from 1 to 4 threads
    doing
    > the same things and went to 800 deadlocks/day overnight.
    > They then increased to 6 threads and doubled the throughput at the cost
    of
    > 8000 deadlocks/day.
    > As soon as they went back to 4 threads, back to 800.
    >
    > Given that the deadlock victims have no data changes to rollback, I
    don't
    > think the occasional deadlocks cost as much as the constant
    inefficiency.
    > I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get
    to
    > the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    > The 5 victims then continue fetching until the next UPDATE "FREE FOR
    ALL"
    > claims 4 more victims... and so on
    >
    > Hopefully they'll reconsider this parallelism strategy.
    >
    > In the meantime, I'm also concerned that the ordering of the rows is
    based
    > on our partitioning scheme instead of an explicit ORDER BY.
    > The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions
    > for the order?
    >
    > I really appreciate all the feedback on this.
    > Thanks!
    > Andre
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: January 23, 2013 5:07 PM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > The first question to ask is the work getting done? If the answer is
    yes,
    > then the urgency of the matter is not critical, and so you can take a
    > meaningful and balanced approach to a resolution.
    >
    > The excessive deadlocks are probably resulting in resource consumption
    in
    > access of what a well behaving application would consume. So, if you are
    > getting the work done then you are just running a little "hot".
    Remember,
    > you don't get deadlocks for free, you have to code them into your
    > application, and so the solution will probably be application based.
    When I
    > design high volume processing systems, especially batch systems as you
    have
    > described, I employ what I call "application controlled parallelism".
    This
    > is where multiple streams are launched in parallel to process the data
    in
    > "logical" partitions. In this way we can avoid all sorts of contention
    and
    > achieve processing rates in excess of 10,000 per second. In contrast,
    you
    > have what I call "application out of control parallelism". I have seen
    this
    > many times before where people use CICS as a way to launch parallel
    batch
    > streams. The same thing happens all the time with locking and other
    > contention.
    >
    > The first thing I would examine is whether or not you can single thread
    > the process. If so, you can see how one thread performs and gauge the
    > impact of the contention. Second, I would see if you can some how
    control
    > the parallelism by getting each thread to work on something different.
    This
    > is probably more difficult to do in CICS then batch. You may need to
    refine
    > your feeder process to somehow split streams to feed to CICS. If you can
    > accomplish something here then that may all that needs to be done. If
    you
    > can't then I like what some people have suggested. However, instead of
    WITH
    > UR on the read I would use FOR UPDATE OF to lock the row immediately,
    but
    > do this in combination with SKIP LOCKED DATA or you will be single
    > threading again. Test this out and see if it reduces contention. it may
    be
    > the easiest thing to do, and result in an "application almost in control
    > parallelism".
    >
    > I hope you find this helpful.
    >
    > Dan
    >
    > On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    >> The high number of deadlocks also concerned me, which is why I'm trying
    > to
    >> come up with a better solution for what they're trying to accomplish.
    >> Unfortunately, they are their own worst enemy since the deadlocks all
    > come
    >> from the same application logic running in parallel on multiple
    threads.
    >> There are no uncommitted inserts or adhoc queries involved.
    >>
    >> I was hoping to get them to commit after the select, but their
    > application
    >> logic only allows them to commit after the update.
    >>
    >> But I am curious about high number of deadlocks:
    >>
    >> - How many deadlocks are too many?
    >>
    >> - Can excessive deadlocks create problems for DB2? (Other
    than
    >> giving me headaches...)
    >>
    >> Thanks!
    >> Andre
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: January 23, 2013 4:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use "WITH UR" only if it's appropriate for your needs. Are
    >> you seeing any lock escalations? What is your commit logic like? Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn't realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    you
    >> create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    >> Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >> Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e>
    >> To: "'[login to unmask email]<mailto:[login to unmask email]>'"<mailto:[login to unmask email]%3e'%22>
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e> wrote:
    >> It's only supposed to skip incompatible locks (I think)... Test Test
    Test...
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> While I'm not sure it's a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for a
    >> situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it's appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    caught
    >> my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here's what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - ...
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don't see any row locks for most Fetches. It appears that row locks
    may
    >> appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    8000
    >> deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I'm not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    Nick Dordea
    [consulting]

    Hi  Andre,

    Have you analyzed the  DB2 log, deadlock messages ? [e.g. DSNT375I, DSNT376I, DSNTU120I, etc.)
    They give you a  lot of information  relative to deadlocks [ plan, correlation,connection, resources, etc} or holders.
    I assume that a DB2 monitor is active. IS IFCID172 activated ? IFCID172  provides info about the participants in deadlocks as well as the survivors . 
    Analyzing the DB2 log + IFCID172 trace/reports vwill help to classify the conditions under which some of deadlock happened :
    case 1 : batch window
    1) collision between batch jobs ? ---> that is a scheduling problem . THye should run serially not in parallel
    2) collision between online transactions with batch jobs? ---> Is is necessary to execute those transactions during batch window?
    3) collision between utility jobs aand non utility jobs/transactions? ---> check the batch schedule

    case 2 : non-batch window
    The same questions as above ... For those jobs are not legit to run during online window daily analyze their schedule, etc

    This analysis will eliminate all deadlocks that happen due to [usually] scheduling errors so they do not affect the application logic.
    After the schedules were enhanced, another deadlok analysis [hopefully] would contain the deadlocks that are due to  the internal logic of applications/programs,  bind parameters, timeouts, etc.  For the long running applications  maybe increasing the commit frequency will help.  Other deadloks may happen due to a conflict between the stability [ isolation ] vs. concurrency [ e.g. RR and RS might be the culprits ].  The non-schedule causes require chaanges in applications and/or bind paarameterrs  even ZPARM paarameters [ in the timeout cases it would be easier to eliminate some timeout values with the value that does not require cheching the time out ... But such thing will impact the performance ...... ]   

    Cheers,

    Nick

     

     

     

     

     



    In Reply to Andre LeBlanc:

    We'll be discussing something similar this afternoon... although there always seems to be some other business rule that complicates the whole process.

    I think I'll have that beer now... :)


    From: Joe Geller [mailto:[login to unmask email]
    Sent: January 24, 2013 11:05 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]


    In regards to Dan's suggestion of having different threads work on different rows; a system I'm supporting had a similar type of process, but with 40 or more threads. They too were getting locking problems. The solution used was to add a column with a thread number. Each inserted row would get the next thread number (rotating from 1 to #threads). The threads that process the rows (to do the update) would only read rows with the corresponding thread #) - completely avoiding the contention.

    The one tricky thing is to handle the ability to change the number of threads. If you increase it, there is no problem. But if you decrease it, then there will be existing rows with a higher thread # than the number of threads, so you need to handle that (perhaps using something like MOD(thread#, #threads))

    Joe

    In Reply to Andre LeBlanc:
    Even though the data is disorganized, working on the data with the earliest Complete-by-Timestamp is critical.
    Some work has to be completed within minutes, while most work has hours to be processed.
    The concern is that we'll miss the tight deadlines without proper ordering.

    Andre

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 24, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    By the way, I'm not criticizing this method of parallelism, but
    emphasizing the fact that there are better ways to do this work. I have a
    customer that sells up to a million products per day. Doesn't seem like
    much in today's world, but they process/track each order, customer, and
    product during the entire life cycle of the sale, shipment, and payment.
    This results in many millions of transactions. Besides customer
    interaction, the entire process is online batch controlled via MQSeries,
    Batch, and CICS. It employs "almost in control parallelism" in that various
    critical components are launched in series. However, once a process is
    started there is no control of the parallelism and we get deadlocks. In
    fact, many thousands of deadlocks each day. It's a fact of a day's work at
    this place, and they accept it. Can they make it better? You bet. However,
    it's an application change and that would cost more money than what the
    deadlocks cost.

    If you are streaming data through this table, which is what it sounds to
    me like what you are doing, then clustering is irrelevant as the data will
    always be disorganized. If you can identify partitions in the driving query
    then you could assign threads to partitions via a control table. If you can
    identify a "type" of record to process then you could do the same thing
    with that column. This would completely eliminate your deadlocks. If none
    of that is possible you could try something crazy like adding a RAND()
    predicate to the query to make your parallelism "completely out of
    control". This could, in theory, reduce your deadlocks by inverse of the
    percentage in your predicate (e.g. AND RAND() < .2 is an 80% reduction).
    You won't eliminate deadlocks, and there might be a slight risk to meeting
    your SLA (but probably not greater than the risk associated with the
    deadlocks in this regard), but it is a one line change in the program and
    you can always increase the parallelism.

    Your current process is like soldiers storming a beach in a single file.
    As one goes down the next climbs over his back and continues the fight.
    Adding the RAND() predicate would scatter the soldiers. You'll still lose a
    few of them, but secure the beach faster and with less casualties.

    hmmm, this sounds like a conversation we should be having after work over
    a beer.

    Cheers,
    Dan

    P.S. I'm patenting the "application completely out of control parallelism"
    model so no one get any smart ideas, eh?

    On Thu, 24 Jan 2013 15:02:51 +0000, LeBlanc, André-ITB
    <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > You've hit the nail on the head Dan, this is precisely "application out
    of
    > control parallelism".
    > They needed higher throughput so they increased from 1 to 4 threads
    doing
    > the same things and went to 800 deadlocks/day overnight.
    > They then increased to 6 threads and doubled the throughput at the cost
    of
    > 8000 deadlocks/day.
    > As soon as they went back to 4 threads, back to 800.
    >
    > Given that the deadlock victims have no data changes to rollback, I
    don't
    > think the occasional deadlocks cost as much as the constant
    inefficiency.
    > I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get
    to
    > the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    > The 5 victims then continue fetching until the next UPDATE "FREE FOR
    ALL"
    > claims 4 more victims... and so on
    >
    > Hopefully they'll reconsider this parallelism strategy.
    >
    > In the meantime, I'm also concerned that the ordering of the rows is
    based
    > on our partitioning scheme instead of an explicit ORDER BY.
    > The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions
    > for the order?
    >
    > I really appreciate all the feedback on this.
    > Thanks!
    > Andre
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: January 23, 2013 5:07 PM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > The first question to ask is the work getting done? If the answer is
    yes,
    > then the urgency of the matter is not critical, and so you can take a
    > meaningful and balanced approach to a resolution.
    >
    > The excessive deadlocks are probably resulting in resource consumption
    in
    > access of what a well behaving application would consume. So, if you are
    > getting the work done then you are just running a little "hot".
    Remember,
    > you don't get deadlocks for free, you have to code them into your
    > application, and so the solution will probably be application based.
    When I
    > design high volume processing systems, especially batch systems as you
    have
    > described, I employ what I call "application controlled parallelism".
    This
    > is where multiple streams are launched in parallel to process the data
    in
    > "logical" partitions. In this way we can avoid all sorts of contention
    and
    > achieve processing rates in excess of 10,000 per second. In contrast,
    you
    > have what I call "application out of control parallelism". I have seen
    this
    > many times before where people use CICS as a way to launch parallel
    batch
    > streams. The same thing happens all the time with locking and other
    > contention.
    >
    > The first thing I would examine is whether or not you can single thread
    > the process. If so, you can see how one thread performs and gauge the
    > impact of the contention. Second, I would see if you can some how
    control
    > the parallelism by getting each thread to work on something different.
    This
    > is probably more difficult to do in CICS then batch. You may need to
    refine
    > your feeder process to somehow split streams to feed to CICS. If you can
    > accomplish something here then that may all that needs to be done. If
    you
    > can't then I like what some people have suggested. However, instead of
    WITH
    > UR on the read I would use FOR UPDATE OF to lock the row immediately,
    but
    > do this in combination with SKIP LOCKED DATA or you will be single
    > threading again. Test this out and see if it reduces contention. it may
    be
    > the easiest thing to do, and result in an "application almost in control
    > parallelism".
    >
    > I hope you find this helpful.
    >
    > Dan
    >
    > On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    >> The high number of deadlocks also concerned me, which is why I'm trying
    > to
    >> come up with a better solution for what they're trying to accomplish.
    >> Unfortunately, they are their own worst enemy since the deadlocks all
    > come
    >> from the same application logic running in parallel on multiple
    threads.
    >> There are no uncommitted inserts or adhoc queries involved.
    >>
    >> I was hoping to get them to commit after the select, but their
    > application
    >> logic only allows them to commit after the update.
    >>
    >> But I am curious about high number of deadlocks:
    >>
    >> - How many deadlocks are too many?
    >>
    >> - Can excessive deadlocks create problems for DB2? (Other
    than
    >> giving me headaches...)
    >>
    >> Thanks!
    >> Andre
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: January 23, 2013 4:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use "WITH UR" only if it's appropriate for your needs. Are
    >> you seeing any lock escalations? What is your commit logic like? Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn't realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    you
    >> create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    >> Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >> Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e>
    >> To: "'[login to unmask email]<" target="_BLANK">mailto:[login to unmask email]>'"<mailto:[login to unmask email]%3e'%22>
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e> wrote:
    >> It's only supposed to skip incompatible locks (I think)... Test Test
    Test...
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> While I'm not sure it's a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for a
    >> situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it's appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    caught
    >> my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here's what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - ...
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don't see any row locks for most Fetches. It appears that row locks
    may
    >> appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    8000
    >> deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I'm not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    Andre LeBlanc
    [CBSA]
    Hi Nick,

    Yes, I have the deadlock report so I know exactly what is causing the deadlock.
    I think it got lost in all the replies, so here it is again:

    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - ...

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit


    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock - waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock - waits on Thread1 U lock

    Thanks, Andre


    From: Nick Dordea [mailto:[login to unmask email]
    Sent: January 24, 2013 12:35 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]


    Hi Andre,

    Have you analyzed the DB2 log, deadlock messages ? [e.g. DSNT375I, DSNT376I, DSNTU120I, etc.)
    They give you a lot of information relative to deadlocks [ plan, correlation,connection, resources, etc} or holders.
    I assume that a DB2 monitor is active. IS IFCID172 activated ? IFCID172 provides info about the participants in deadlocks as well as the survivors .
    Analyzing the DB2 log + IFCID172 trace/reports vwill help to classify the conditions under which some of deadlock happened :
    case 1 : batch window
    1) collision between batch jobs ? ---> that is a scheduling problem . THye should run serially not in parallel
    2) collision between online transactions with batch jobs? ---> Is is necessary to execute those transactions during batch window?
    3) collision between utility jobs aand non utility jobs/transactions? ---> check the batch schedule

    case 2 : non-batch window
    The same questions as above ... For those jobs are not legit to run during online window daily analyze their schedule, etc

    This analysis will eliminate all deadlocks that happen due to [usually] scheduling errors so they do not affect the application logic.
    After the schedules were enhanced, another deadlok analysis [hopefully] would contain the deadlocks that are due to the internal logic of applications/programs, bind parameters, timeouts, etc. For the long running applications maybe increasing the commit frequency will help. Other deadloks may happen due to a conflict between the stability [ isolation ] vs. concurrency [ e.g. RR and RS might be the culprits ]. The non-schedule causes require chaanges in applications and/or bind paarameterrs even ZPARM paarameters [ in the timeout cases it would be easier to eliminate some timeout values with the value that does not require cheching the time out ... But such thing will impact the performance ...... ]

    Cheers,

    Nick












    In Reply to Andre LeBlanc:
    We'll be discussing something similar this afternoon... although there always seems to be some other business rule that complicates the whole process.

    I think I'll have that beer now... :)


    From: Joe Geller [mailto:[login to unmask email]
    Sent: January 24, 2013 11:05 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]


    In regards to Dan's suggestion of having different threads work on different rows; a system I'm supporting had a similar type of process, but with 40 or more threads. They too were getting locking problems. The solution used was to add a column with a thread number. Each inserted row would get the next thread number (rotating from 1 to #threads). The threads that process the rows (to do the update) would only read rows with the corresponding thread #) - completely avoiding the contention.

    The one tricky thing is to handle the ability to change the number of threads. If you increase it, there is no problem. But if you decrease it, then there will be existing rows with a higher thread # than the number of threads, so you need to handle that (perhaps using something like MOD(thread#, #threads))

    Joe

    In Reply to Andre LeBlanc:
    Even though the data is disorganized, working on the data with the earliest Complete-by-Timestamp is critical.
    Some work has to be completed within minutes, while most work has hours to be processed.
    The concern is that we'll miss the tight deadlines without proper ordering.

    Andre

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 24, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    By the way, I'm not criticizing this method of parallelism, but
    emphasizing the fact that there are better ways to do this work. I have a
    customer that sells up to a million products per day. Doesn't seem like
    much in today's world, but they process/track each order, customer, and
    product during the entire life cycle of the sale, shipment, and payment.
    This results in many millions of transactions. Besides customer
    interaction, the entire process is online batch controlled via MQSeries,
    Batch, and CICS. It employs "almost in control parallelism" in that various
    critical components are launched in series. However, once a process is
    started there is no control of the parallelism and we get deadlocks. In
    fact, many thousands of deadlocks each day. It's a fact of a day's work at
    this place, and they accept it. Can they make it better? You bet. However,
    it's an application change and that would cost more money than what the
    deadlocks cost.

    If you are streaming data through this table, which is what it sounds to
    me like what you are doing, then clustering is irrelevant as the data will
    always be disorganized. If you can identify partitions in the driving query
    then you could assign threads to partitions via a control table. If you can
    identify a "type" of record to process then you could do the same thing
    with that column. This would completely eliminate your deadlocks. If none
    of that is possible you could try something crazy like adding a RAND()
    predicate to the query to make your parallelism "completely out of
    control". This could, in theory, reduce your deadlocks by inverse of the
    percentage in your predicate (e.g. AND RAND() < .2 is an 80% reduction).
    You won't eliminate deadlocks, and there might be a slight risk to meeting
    your SLA (but probably not greater than the risk associated with the
    deadlocks in this regard), but it is a one line change in the program and
    you can always increase the parallelism.

    Your current process is like soldiers storming a beach in a single file.
    As one goes down the next climbs over his back and continues the fight.
    Adding the RAND() predicate would scatter the soldiers. You'll still lose a
    few of them, but secure the beach faster and with less casualties.

    hmmm, this sounds like a conversation we should be having after work over
    a beer.

    Cheers,
    Dan

    P.S. I'm patenting the "application completely out of control parallelism"
    model so no one get any smart ideas, eh?

    On Thu, 24 Jan 2013 15:02:51 +0000, LeBlanc, André-ITB
    <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e> wrote:
    > You've hit the nail on the head Dan, this is precisely "application out
    of
    > control parallelism".
    > They needed higher throughput so they increased from 1 to 4 threads
    doing
    > the same things and went to 800 deadlocks/day overnight.
    > They then increased to 6 threads and doubled the throughput at the cost
    of
    > 8000 deadlocks/day.
    > As soon as they went back to 4 threads, back to 800.
    >
    > Given that the deadlock victims have no data changes to rollback, I
    don't
    > think the occasional deadlocks cost as much as the constant
    inefficiency.
    > I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get
    to
    > the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    > The 5 victims then continue fetching until the next UPDATE "FREE FOR
    ALL"
    > claims 4 more victims... and so on
    >
    > Hopefully they'll reconsider this parallelism strategy.
    >
    > In the meantime, I'm also concerned that the ordering of the rows is
    based
    > on our partitioning scheme instead of an explicit ORDER BY.
    > The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions
    > for the order?
    >
    > I really appreciate all the feedback on this.
    > Thanks!
    > Andre
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: January 23, 2013 5:07 PM
    > To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > The first question to ask is the work getting done? If the answer is
    yes,
    > then the urgency of the matter is not critical, and so you can take a
    > meaningful and balanced approach to a resolution.
    >
    > The excessive deadlocks are probably resulting in resource consumption
    in
    > access of what a well behaving application would consume. So, if you are
    > getting the work done then you are just running a little "hot".
    Remember,
    > you don't get deadlocks for free, you have to code them into your
    > application, and so the solution will probably be application based.
    When I
    > design high volume processing systems, especially batch systems as you
    have
    > described, I employ what I call "application controlled parallelism".
    This
    > is where multiple streams are launched in parallel to process the data
    in
    > "logical" partitions. In this way we can avoid all sorts of contention
    and
    > achieve processing rates in excess of 10,000 per second. In contrast,
    you
    > have what I call "application out of control parallelism". I have seen
    this
    > many times before where people use CICS as a way to launch parallel
    batch
    > streams. The same thing happens all the time with locking and other
    > contention.
    >
    > The first thing I would examine is whether or not you can single thread
    > the process. If so, you can see how one thread performs and gauge the
    > impact of the contention. Second, I would see if you can some how
    control
    > the parallelism by getting each thread to work on something different.
    This
    > is probably more difficult to do in CICS then batch. You may need to
    refine
    > your feeder process to somehow split streams to feed to CICS. If you can
    > accomplish something here then that may all that needs to be done. If
    you
    > can't then I like what some people have suggested. However, instead of
    WITH
    > UR on the read I would use FOR UPDATE OF to lock the row immediately,
    but
    > do this in combination with SKIP LOCKED DATA or you will be single
    > threading again. Test this out and see if it reduces contention. it may
    be
    > the easiest thing to do, and result in an "application almost in control
    > parallelism".
    >
    > I hope you find this helpful.
    >
    > Dan
    >
    > On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e> wrote:
    >> The high number of deadlocks also concerned me, which is why I'm trying
    > to
    >> come up with a better solution for what they're trying to accomplish.
    >> Unfortunately, they are their own worst enemy since the deadlocks all
    > come
    >> from the same application logic running in parallel on multiple
    threads.
    >> There are no uncommitted inserts or adhoc queries involved.
    >>
    >> I was hoping to get them to commit after the select, but their
    > application
    >> logic only allows them to commit after the update.
    >>
    >> But I am curious about high number of deadlocks:
    >>
    >> - How many deadlocks are too many?
    >>
    >> - Can excessive deadlocks create problems for DB2? (Other
    than
    >> giving me headaches...)
    >>
    >> Thanks!
    >> Andre
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: January 23, 2013 4:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use "WITH UR" only if it's appropriate for your needs. Are
    >> you seeing any lock escalations? What is your commit logic like? Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn't realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    you
    >> create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    >> Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >> Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e>
    >> To: "'[login to unmask email]<mailto:[login to unmask email]>'"<mailto:[login to unmask email]%3e'%22><mailto:[login to unmask email]%3e'%22%3cmailto:[login to unmask email]%3e'%22%3e>
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e> wrote:
    >> It's only supposed to skip incompatible locks (I think)... Test Test
    Test...
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'><mailto:[login to unmask email]%3e'%3cmailto:[login to unmask email]%3e'%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> While I'm not sure it's a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for a
    >> situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it's appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'><mailto:[login to unmask email]%3e'%3cmailto:[login to unmask email]%3e'%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    caught
    >> my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here's what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - ...
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don't see any row locks for most Fetches. It appears that row locks
    may
    >> appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    8000
    >> deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I'm not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    Dee Reins
    [Centurylink]
    We have increased MAX LOCKS on table spaces where appropriate. In a 40 million row table 100,000 for max locks works fine I our environment. I believe MAX LOCKS should be a number relative to the size of the table.
    Another note;
    This is what I found about “With UR”, to me this is why it is working in our environment.
    Uncommitted Read: The Uncommitted Read (UR) isolation level, also known as "dirty read," is the lowest level of isolation supported by DB2. It can be used to access uncommitted data changes of other applications. For example, an application using the Uncommitted Read isolation level will return all of the matching rows for the query, even if that data is in the process of being modified and may not be committed to the database. You need to be aware that if you are using Uncommitted Read, two identical queries may get different results, even if they are issued within a unit of work, since other concurrent applications can change or modify those rows that the first query retrieves.
    Uncommitted Read transactions will hold very few locks. Thus they are not likely to wait for other transactions to release locks. If you are accessing read-only tables or it is acceptable for the application to retrieve uncommitted data updated by another application, use this isolation level, because it is most preferable in terms of performance.




    "Leadership and learning are indispensable to each other." - John Fitzgerald Kennedy

    360 905-7343 Jan 31 new number is 360 905-7943
    360 448-8623 (c)

    From: David Simpson [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 4:05 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Let me (re)-understand the problem. Each concurrent task retrieves a row to begin processing and attempts to lock it down by issuing an update statement. If this is true WITH UR may only shift the problem to the update and cause excessive waits rather than timeouts. Using WITH UR both competing tasks will still attempt to do the same piece of work. I believe SKIP LOCKED DATA offers the best chance for success here since once TASK A issues its update, TASK B will not attempt to read the row.

    However, if the situation is more complex than my simple understanding, of course there could be other issues.

    From: Reins, Dee D [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 5:06 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    We have data sharing, and had problems with locking. We carefully added row level locking to the tables we needed to, and that resolved our issue. Row level locking should not be a problem in a single DB2 environment. I suggest picking a table with the most locks, make it row level locking. The next table will bubble to the top, repeat the process. Doing this with “WITH UR”, should help a lot.





    "Leadership and learning are indispensable to each other." - John Fitzgerald Kennedy

    360 905-7343 Jan 31 new number is 360 905-7943
    360 448-8623 (c)

    From: Campbell, Wayne [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 1:42 PM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Remember to use “WITH UR” only if it’s appropriate for your needs. Are you seeing any lock escalations? What is your commit logic like? Do you commit after select statement? This got us a little while back, the developers didn’t realize that selects took locks and left them hanging.
    Wayne Campbell
    DB2 DBA
    Administrative Office of the Courts
    (360) 705-5268
    Email: [login to unmask email]<mailto:[login to unmask email]>

    From: Charles Browm [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    More than 500 within a day -- never heard of that many deadlocks not even within a month period. Is there something fundamentally wrong -- you bet!! For a starter, i would suggest you update all your Ad hoc READs to use " ...WITH UR".
    Next increase your zparm timeout parm Take advantage of RCT parm ROLBE=NO. And finally take advantage of some of the lock avoidance parms available in v9 such as SKIPUNCI -- someone has already mentioned. BTW your bind parms look good.

    If you're still getting more than 20 a day -- I'll suggest its time you create anther SUBSYS. Some applications just can't coexist.

    Thx and best,

    Sent from my iPhone

    On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    In this case, the update does not move the row to another partition.

    From: Walter Janißen [mailto:[login to unmask email]
    Sent: January 23, 2013 10:27 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    But that is no longer a problem as in some earlier DB2 versions.

    Mit freundlichen Grüßen
    Walter Janißen

    ITERGO Informationstechnologie GmbH
    Anwendungsentwicklung
    Technische Anwendungsarchitektur
    Victoriaplatz 2
    D-40198 Düsseldorf
    [login to unmask email]<mailto:[login to unmask email]>

    ITERGO Informationstechnologie GmbH
    Vorsitzender des Aufsichtsrats: Christian Diedrich
    Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael Regauer
    Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996


    ________________________________
    Von: Basivi Inaganti [mailto:[login to unmask email]
    Gesendet: Mittwoch, 23. Januar 2013 16:19
    An: [login to unmask email]<mailto:[login to unmask email]>
    Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    Andre,

    By any chance, is the update causing the rows to move to new partition?

    Thanks,
    ______________________________________________________________________
    [cid:[login to unmask email]



    From: "LeBlanc,André-ITB" <[login to unmask email]<mailto:[login to unmask email]>>
    To: "'[login to unmask email]<mailto:[login to unmask email]>'" <[login to unmask email]<mailto:[login to unmask email]>>
    Date: 01/23/2013 10:01 AM
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]
    ________________________________



    You are correct Charles, these are CICS COBOL packages.
    Here are the bind parameters:

    RELEASE ( COMMIT ) ISOLATION ( CS )
    VALIDATE ( BIND ) EXPLAIN ( YES )
    DYNAMICRULES ( )
    DEGREE ( 1 ) CURRENTDATA ( NO )
    KEEPDYNAMIC ( NO )
    DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    PATHDEFAULT
    IMMEDWRITE ( NO ) ENCODING ( 37 )


    From: Charles Browm [mailto:[login to unmask email]
    Sent: January 23, 2013 9:48 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre
    What level of db2 concurrency are you using? My guess, these are static db2 packages -- please advise because I couldn't find these pertinent info in your problem description.
    Best and Thx !!!

    Sent from my iPhone

    On Jan 23, 2013, at 8:30 AM, David Simpson <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    It’s only supposed to skip incompatible locks (I think)… Test Test Test…

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:30 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I thought of that, but since it can get the S lock, would it skip the row in the fetch?

    Thanks,
    André LeBlanc

    From: David Simpson [mailto:[login to unmask email]
    Sent: January 23, 2013 9:28 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    While I’m not sure it’s a good fit you could consider the SKIP LOCKED DATA clause on your cursor select in DB2 9. This feature was intended for a situation like yours. If the cursor encounters a locked row it will skip it without waiting. I would certainly test this heavily in your environment to see if it’s appropriate.

    ______________________________________________________________________________
    David Simpson | Senior Technical Advisor | Themis Education
    123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 | [login to unmask email]<mailto:[login to unmask email]>
    For more information about Themis, visit www.themisinc.comhttp://www.themisinc.com

    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Wednesday, January 23, 2013 8:23 AM
    To: '[login to unmask email]<mailto:[login to unmask email]>'
    Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Esteemed Listers, I can really use your help.

    When we started getting 800 deadlocks a day on one tablespace, it caught my attention.
    And then it climbed to 8000 deadlocks per day!

    Here’s what I found:
    - Our application needs to process data by a given Timestamp, assigned based on priority.
    - Some data has up to 4 hours to be processed, others must be processed within 2 minutes
    - Some rows may be related to other rows (A few or a few thousand) in a separate relationship table
    - All related data must be processed in the sequence it was captured.
    - Tablespace has 120 parts, partitioned and clustered by day/hour, with ROW level locking.
    - At any given time, 1 partition is very busy, 2 partitions are somewhat busy, and all other partitions are empty.
    - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH HOLD so that data inserted with earlier timestamp is picked up immediately
    - Cursor has no ORDER BY. Access to earliest timestamp is based on DB2 accessing earlier partitions first
    - Multiple threads compete to process data in this table
    - When a thread finds a candidate row, it claims it by issuing a separate UPDATE outside of cursor. (not where current of cursor)

    The transaction involved in the deadlocks is as follows:
    - Open Cursor
    - Fetch1
    - Select1 (if earlier related rows exist, fetch next row)
    - Fetch2
    - Select2
    - …
    - Fetch44
    - Select44 (rc=+100) no earlier related row found so thread will try to claim row
    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)
    - Commit

    Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1- The first thread to get to the update gets U lock on row1.
    2- Thread2 grab a S lock on row1 during Fetch44.
    3- Thread1 wants to change its U lock to X lock ?C waits on Thread2 S lock
    4- Thread2 issues UPDATE and tries to get U lock ?C waits on Thread1 U lock

    I have an index that gives the Cursor INDEX ONLY access, so I actually don’t see any row locks for most Fetches. It appears that row locks may appear when another thread attempts an UPDATE.

    We see 800 deadlocks a day when running 4 competing threads.
    By going to 6 threads they doubled the throughput, and increased to 8000 deadlocks a day.

    Everything runs in milliseconds since these pages are always in Buffer Pool so overall response time is within the SLA.
    There is retry logic, so the end user is not that concerned about the deadlocks.

    I’m not sure what my next step should be, so any suggestions you may have are greatly appreciated.

    Cheers!
    Andr?? LeBlanc


    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    -----End Original Message-----
    Nick Dordea
    [consulting]

    Hi Andre,

    Thanks for the quick reply and clarification.

    Based on your description ...
    "Deadlock occurs because multiple threads are all executing this same transaction at the same time.
    1-      The first thread to get to the update gets U lock on row1.
    2-      Thread2 grab a S lock on row1 during Fetch44.
    3-      Thread1 wants to change its U lock to X lock – waits on Thread2 S lock
    4-      Thread2 issues UPDATE and tries to get U lock – waits on Thread1 U lock"

    As SG24-4725-01 [ DB2 9 for z/OS:Resource Serialization and Concurrency Control ]

    -- "While a page or row is U locked, if any other
    concurrent process wants to do an update or delete, it must first acquire a U or X lock, but has
    to wait and eventually either the U or X lock is acquired or a timeout occurs."

    -- S and U locks are "gross lock states" that limits the concurrency and discusses about SIX lock
    that is not as restrictive as any of U or S lock.   

    The interesting article : "The Mystery of the DB2 SIX Lock" by Bonnie Baker
    ttp://www.toadworld.com/Newsletter/TWPIPELINEMarch2009/PIPEMarch09DB2/tabid/563/Default.aspx
    discusses the SIX locks that may solve your situation 

    Cheers,

    Nick

     

     

     

     


       

     

     



    In Reply to Andre LeBlanc:

    Hi Nick,

    Yes, I have the deadlock report so I know exactly what is causing the deadlock.
    I think it got lost in all the replies, so here it is again:

    The transaction involved in the deadlocks is as follows:

    - Open Cursor

    - Fetch1

    - Select1 (if earlier related rows exist, fetch next row)

    - Fetch2

    - Select2

    - ...

    - Fetch44

    - Select44 (rc=+100) no earlier related row found so thread will try to claim row

    - Update row using values from Fetch44 (will get +100 if row already updated by another thread)

    - Commit


    Deadlock occurs because multiple threads are all executing this same transaction at the same time.

    1- The first thread to get to the update gets U lock on row1.

    2- Thread2 grab a S lock on row1 during Fetch44.

    3- Thread1 wants to change its U lock to X lock - waits on Thread2 S lock

    4- Thread2 issues UPDATE and tries to get U lock - waits on Thread1 U lock

    Thanks, Andre


    From: Nick Dordea [mailto:[login to unmask email]
    Sent: January 24, 2013 12:35 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]


    Hi Andre,

    Have you analyzed the DB2 log, deadlock messages ? [e.g. DSNT375I, DSNT376I, DSNTU120I, etc.)
    They give you a lot of information relative to deadlocks [ plan, correlation,connection, resources, etc} or holders.
    I assume that a DB2 monitor is active. IS IFCID172 activated ? IFCID172 provides info about the participants in deadlocks as well as the survivors .
    Analyzing the DB2 log + IFCID172 trace/reports vwill help to classify the conditions under which some of deadlock happened :
    case 1 : batch window
    1) collision between batch jobs ? ---> that is a scheduling problem . THye should run serially not in parallel
    2) collision between online transactions with batch jobs? ---> Is is necessary to execute those transactions during batch window?
    3) collision between utility jobs aand non utility jobs/transactions? ---> check the batch schedule

    case 2 : non-batch window
    The same questions as above ... For those jobs are not legit to run during online window daily analyze their schedule, etc

    This analysis will eliminate all deadlocks that happen due to [usually] scheduling errors so they do not affect the application logic.
    After the schedules were enhanced, another deadlok analysis [hopefully] would contain the deadlocks that are due to the internal logic of applications/programs, bind parameters, timeouts, etc. For the long running applications maybe increasing the commit frequency will help. Other deadloks may happen due to a conflict between the stability [ isolation ] vs. concurrency [ e.g. RR and RS might be the culprits ]. The non-schedule causes require chaanges in applications and/or bind paarameterrs even ZPARM paarameters [ in the timeout cases it would be easier to eliminate some timeout values with the value that does not require cheching the time out ... But such thing will impact the performance ...... ]

    Cheers,

    Nick












    In Reply to Andre LeBlanc:
    We'll be discussing something similar this afternoon... although there always seems to be some other business rule that complicates the whole process.

    I think I'll have that beer now... :)


    From: Joe Geller [mailto:[login to unmask email]
    Sent: January 24, 2013 11:05 AM
    To: [login to unmask email]<mailto:[login to unmask email]>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]


    In regards to Dan's suggestion of having different threads work on different rows; a system I'm supporting had a similar type of process, but with 40 or more threads. They too were getting locking problems. The solution used was to add a column with a thread number. Each inserted row would get the next thread number (rotating from 1 to #threads). The threads that process the rows (to do the update) would only read rows with the corresponding thread #) - completely avoiding the contention.

    The one tricky thing is to handle the ability to change the number of threads. If you increase it, there is no problem. But if you decrease it, then there will be existing rows with a higher thread # than the number of threads, so you need to handle that (perhaps using something like MOD(thread#, #threads))

    Joe

    In Reply to Andre LeBlanc:
    Even though the data is disorganized, working on the data with the earliest Complete-by-Timestamp is critical.
    Some work has to be completed within minutes, while most work has hours to be processed.
    The concern is that we'll miss the tight deadlines without proper ordering.

    Andre

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: January 24, 2013 10:33 AM
    To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    By the way, I'm not criticizing this method of parallelism, but
    emphasizing the fact that there are better ways to do this work. I have a
    customer that sells up to a million products per day. Doesn't seem like
    much in today's world, but they process/track each order, customer, and
    product during the entire life cycle of the sale, shipment, and payment.
    This results in many millions of transactions. Besides customer
    interaction, the entire process is online batch controlled via MQSeries,
    Batch, and CICS. It employs "almost in control parallelism" in that various
    critical components are launched in series. However, once a process is
    started there is no control of the parallelism and we get deadlocks. In
    fact, many thousands of deadlocks each day. It's a fact of a day's work at
    this place, and they accept it. Can they make it better? You bet. However,
    it's an application change and that would cost more money than what the
    deadlocks cost.

    If you are streaming data through this table, which is what it sounds to
    me like what you are doing, then clustering is irrelevant as the data will
    always be disorganized. If you can identify partitions in the driving query
    then you could assign threads to partitions via a control table. If you can
    identify a "type" of record to process then you could do the same thing
    with that column. This would completely eliminate your deadlocks. If none
    of that is possible you could try something crazy like adding a RAND()
    predicate to the query to make your parallelism "completely out of
    control". This could, in theory, reduce your deadlocks by inverse of the
    percentage in your predicate (e.g. AND RAND() < .2 is an 80% reduction).
    You won't eliminate deadlocks, and there might be a slight risk to meeting
    your SLA (but probably not greater than the risk associated with the
    deadlocks in this regard), but it is a one line change in the program and
    you can always increase the parallelism.

    Your current process is like soldiers storming a beach in a single file.
    As one goes down the next climbs over his back and continues the fight.
    Adding the RAND() predicate would scatter the soldiers. You'll still lose a
    few of them, but secure the beach faster and with less casualties.

    hmmm, this sounds like a conversation we should be having after work over
    a beer.

    Cheers,
    Dan

    P.S. I'm patenting the "application completely out of control parallelism"
    model so no one get any smart ideas, eh?

    On Thu, 24 Jan 2013 15:02:51 +0000, LeBlanc, André-ITB
    <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e> wrote:
    > You've hit the nail on the head Dan, this is precisely "application out
    of
    > control parallelism".
    > They needed higher throughput so they increased from 1 to 4 threads
    doing
    > the same things and went to 800 deadlocks/day overnight.
    > They then increased to 6 threads and doubled the throughput at the cost
    of
    > 8000 deadlocks/day.
    > As soon as they went back to 4 threads, back to 800.
    >
    > Given that the deadlock victims have no data changes to rollback, I
    don't
    > think the occasional deadlocks cost as much as the constant
    inefficiency.
    > I've seen 6 threads all doing the same 200+ FETCHES and SELECTS to get
    to
    > the SEARCHED UPDATE, and all but one get a DEADLOCK or ROW NOT FOUND.
    > The 5 victims then continue fetching until the next UPDATE "FREE FOR
    ALL"
    > claims 4 more victims... and so on
    >
    > Hopefully they'll reconsider this parallelism strategy.
    >
    > In the meantime, I'm also concerned that the ordering of the rows is
    based
    > on our partitioning scheme instead of an explicit ORDER BY.
    > The ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions
    > for the order?
    >
    > I really appreciate all the feedback on this.
    > Thanks!
    > Andre
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: January 23, 2013 5:07 PM
    > To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > The first question to ask is the work getting done? If the answer is
    yes,
    > then the urgency of the matter is not critical, and so you can take a
    > meaningful and balanced approach to a resolution.
    >
    > The excessive deadlocks are probably resulting in resource consumption
    in
    > access of what a well behaving application would consume. So, if you are
    > getting the work done then you are just running a little "hot".
    Remember,
    > you don't get deadlocks for free, you have to code them into your
    > application, and so the solution will probably be application based.
    When I
    > design high volume processing systems, especially batch systems as you
    have
    > described, I employ what I call "application controlled parallelism".
    This
    > is where multiple streams are launched in parallel to process the data
    in
    > "logical" partitions. In this way we can avoid all sorts of contention
    and
    > achieve processing rates in excess of 10,000 per second. In contrast,
    you
    > have what I call "application out of control parallelism". I have seen
    this
    > many times before where people use CICS as a way to launch parallel
    batch
    > streams. The same thing happens all the time with locking and other
    > contention.
    >
    > The first thing I would examine is whether or not you can single thread
    > the process. If so, you can see how one thread performs and gauge the
    > impact of the contention. Second, I would see if you can some how
    control
    > the parallelism by getting each thread to work on something different.
    This
    > is probably more difficult to do in CICS then batch. You may need to
    refine
    > your feeder process to somehow split streams to feed to CICS. If you can
    > accomplish something here then that may all that needs to be done. If
    you
    > can't then I like what some people have suggested. However, instead of
    WITH
    > UR on the read I would use FOR UPDATE OF to lock the row immediately,
    but
    > do this in combination with SKIP LOCKED DATA or you will be single
    > threading again. Test this out and see if it reduces contention. it may
    be
    > the easiest thing to do, and result in an "application almost in control
    > parallelism".
    >
    > I hope you find this helpful.
    >
    > Dan
    >
    > On Wed, 23 Jan 2013 21:50:37 +0000, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e> wrote:
    >> The high number of deadlocks also concerned me, which is why I'm trying
    > to
    >> come up with a better solution for what they're trying to accomplish.
    >> Unfortunately, they are their own worst enemy since the deadlocks all
    > come
    >> from the same application logic running in parallel on multiple
    threads.
    >> There are no uncommitted inserts or adhoc queries involved.
    >>
    >> I was hoping to get them to commit after the select, but their
    > application
    >> logic only allows them to commit after the update.
    >>
    >> But I am curious about high number of deadlocks:
    >>
    >> - How many deadlocks are too many?
    >>
    >> - Can excessive deadlocks create problems for DB2? (Other
    than
    >> giving me headaches...)
    >>
    >> Thanks!
    >> Andre
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: January 23, 2013 4:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use "WITH UR" only if it's appropriate for your needs. Are
    >> you seeing any lock escalations? What is your commit logic like? Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn't realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    you
    >> create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    >> Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >> Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e>
    >> To: "'[login to unmask email]<'"" target="_BLANK">mailto:[login to unmask email]>'"<mailto:[login to unmask email]%3e'%22><mailto:[login to unmask email]%3e'%22%3cmailto:[login to unmask email]%3e'%22%3e>
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>><mailto:[login to unmask email]%3e%3e><mailto:[login to unmask email]%3e%3e%3cmailto:[login to unmask email]%3e%3e%3e> wrote:
    >> It's only supposed to skip incompatible locks (I think)... Test Test
    Test...
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'><mailto:[login to unmask email]%3e'%3cmailto:[login to unmask email]%3e'%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> While I'm not sure it's a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for a
    >> situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it's appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]><mailto:[login to unmask email]%3e><mailto:[login to unmask email]%3e%3cmailto:[login to unmask email]%3e%3e>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'<mailto:[login to unmask email]%3e'><mailto:[login to unmask email]%3e'%3cmailto:[login to unmask email]%3e'%3e>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    >> z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    caught
    >> my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here's what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - ...
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don't see any row locks for most Fetches. It appears that row locks
    may
    >> appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    8000
    >> deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I'm not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----

    -----End Original Message-----
    Daniel Luksetich
    [DanL Database Consulting]
    In this particular situation the use of UR will have to go hand in hand
    with an optimistic locking strategy (i.e. checking timestamps on updates).
    Cheers,
    Dan

    On Thu, 24 Jan 2013 18:18:17 +0000, "Reins, Dee D"
    <[login to unmask email]> wrote:
    > We have increased MAX LOCKS on table spaces where appropriate. In a 40
    > million row table 100,000 for max locks works fine I our environment.
    I
    > believe MAX LOCKS should be a number relative to the size of the table.
    > Another note;
    > This is what I found about “With UR”, to me this is why it is working
    in
    > our environment.
    > Uncommitted Read: The Uncommitted Read (UR) isolation level, also known
    as
    > "dirty read," is the lowest level of isolation supported by DB2. It can
    be
    > used to access uncommitted data changes of other applications. For
    example,
    > an application using the Uncommitted Read isolation level will return
    all
    > of the matching rows for the query, even if that data is in the process
    of
    > being modified and may not be committed to the database. You need to be
    > aware that if you are using Uncommitted Read, two identical queries may
    get
    > different results, even if they are issued within a unit of work, since
    > other concurrent applications can change or modify those rows that the
    > first query retrieves.
    > Uncommitted Read transactions will hold very few locks. Thus they are
    not
    > likely to wait for other transactions to release locks. If you are
    > accessing read-only tables or it is acceptable for the application to
    > retrieve uncommitted data updated by another application, use this
    > isolation level, because it is most preferable in terms of performance.
    >
    >
    >
    >
    > "Leadership and learning are indispensable to each other." - John
    > Fitzgerald Kennedy
    >
    > 360 905-7343 Jan 31 new number is 360 905-7943
    > 360 448-8623 (c)
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 4:05 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Let me (re)-understand the problem. Each concurrent task retrieves a
    row
    > to begin processing and attempts to lock it down by issuing an update
    > statement. If this is true WITH UR may only shift the problem to the
    > update and cause excessive waits rather than timeouts. Using WITH UR
    both
    > competing tasks will still attempt to do the same piece of work. I
    believe
    > SKIP LOCKED DATA offers the best chance for success here since once TASK
    A
    > issues its update, TASK B will not attempt to read the row.
    >
    > However, if the situation is more complex than my simple understanding,
    of
    > course there could be other issues.
    >
    > From: Reins, Dee D [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 5:06 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > We have data sharing, and had problems with locking. We carefully added
    > row level locking to the tables we needed to, and that resolved our
    issue.
    > Row level locking should not be a problem in a single DB2 environment.

    > I suggest picking a table with the most locks, make it row level
    locking.
    > The next table will bubble to the top, repeat the process. Doing this
    > with “WITH UR”, should help a lot.
    >
    >
    >
    >
    >
    > "Leadership and learning are indispensable to each other." - John
    > Fitzgerald Kennedy
    >
    > 360 905-7343 Jan 31 new number is 360 905-7943
    > 360 448-8623 (c)
    >
    > From: Campbell, Wayne [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 1:42 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Remember to use “WITH UR” only if it’s appropriate for your needs. Are
    > you seeing any lock escalations? What is your commit logic like? Do
    you
    > commit after select statement? This got us a little while back, the
    > developers didn’t realize that selects took locks and left them hanging.
    > Wayne Campbell
    > DB2 DBA
    > Administrative Office of the Courts
    > (360) 705-5268
    > Email: [login to unmask email]<mailto:[login to unmask email]>
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 10:33 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > More than 500 within a day -- never heard of that many deadlocks not
    even
    > within a month period. Is there something fundamentally wrong -- you
    bet!!
    > For a starter, i would suggest you update all your Ad hoc READs to use
    "
    > ...WITH UR".
    > Next increase your zparm timeout parm Take advantage of RCT parm
    > ROLBE=NO. And finally take advantage of some of the lock avoidance
    parms
    > available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    your
    > bind parms look good.
    >
    > If you're still getting more than 20 a day -- I'll suggest its time you
    > create anther SUBSYS. Some applications just can't coexist.
    >
    > Thx and best,
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > wrote:
    > In this case, the update does not move the row to another partition.
    >
    > From: Walter Janißen [mailto:[login to unmask email]
    > Sent: January 23, 2013 10:27 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > But that is no longer a problem as in some earlier DB2 versions.
    >
    > Mit freundlichen Grüßen
    > Walter Janißen
    >
    > ITERGO Informationstechnologie GmbH
    > Anwendungsentwicklung
    > Technische Anwendungsarchitektur
    > Victoriaplatz 2
    > D-40198 Düsseldorf
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > ITERGO Informationstechnologie GmbH
    > Vorsitzender des Aufsichtsrats: Christian Diedrich
    > Geschäftsführung: Dr. Bettina Anders (Vorsitzende),
    > Lothar Engelke, Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    Regauer
    > Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >
    >
    > ________________________________
    > Von: Basivi Inaganti [mailto:[login to unmask email]
    > Gesendet: Mittwoch, 23. Januar 2013 16:19
    > An: [login to unmask email]<mailto:[login to unmask email]>
    > Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    > Andre,
    >
    > By any chance, is the update causing the rows to move to new partition?
    >
    > Thanks,
    > ______________________________________________________________________
    > [cid:[login to unmask email]
    >
    >
    >
    > From: "LeBlanc,André-ITB"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > Date: 01/23/2013 10:01 AM
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    my
    > [DB2 z/OS v9]
    > ________________________________
    >
    >
    >
    > You are correct Charles, these are CICS COBOL packages.
    > Here are the bind parameters:
    >
    > RELEASE ( COMMIT ) ISOLATION ( CS )
    > VALIDATE ( BIND ) EXPLAIN ( YES )
    > DYNAMICRULES ( )
    > DEGREE ( 1 ) CURRENTDATA ( NO )
    > KEEPDYNAMIC ( NO )
    > DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    > PATHDEFAULT
    > IMMEDWRITE ( NO ) ENCODING ( 37 )
    >
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:48 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > Andre
    > What level of db2 concurrency are you using? My guess, these are static
    > db2 packages -- please advise because I couldn't find these pertinent
    info
    > in your problem description.
    > Best and Thx !!!
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 8:30 AM, David Simpson
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > It’s only supposed to skip incompatible locks (I think)… Test Test Test…
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:30 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > I thought of that, but since it can get the S lock, would it skip the
    row
    > in the fetch?
    >
    > Thanks,
    > André LeBlanc
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:28 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    DATA
    > clause on your cursor select in DB2 9. This feature was intended for a
    > situation like yours. If the cursor encounters a locked row it will
    skip
    > it without waiting. I would certainly test this heavily in your
    > environment to see if it’s appropriate.
    >
    >
    ______________________________________________________________________________
    > David Simpson | Senior Technical Advisor | Themis Education
    > 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    > [login to unmask email]<mailto:[login to unmask email]>
    > For more information about Themis, visit
    > www.themisinc.comhttp://www.themisinc.com
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:23 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my [DB2
    > z/OS v9]
    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it caught
    > my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks may
    > appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to 8000
    > deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    > -----End Original Message-----
    >
    > -----End Original Message-----
    David Simpson
    [Themis Inc.]
    Also true if SKIP LOCKED DATA is employed although I think SKIP LOCKED DATA will result in less contention than UR in this case since it will skip over any items that have already been UPDATEd by another process and won't even see them. UR will see them and then fail later.

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: Thursday, January 24, 2013 3:25 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    In this particular situation the use of UR will have to go hand in hand with an optimistic locking strategy (i.e. checking timestamps on updates).
    Cheers,
    Dan

    On Thu, 24 Jan 2013 18:18:17 +0000, "Reins, Dee D"
    <[login to unmask email]> wrote:
    > We have increased MAX LOCKS on table spaces where appropriate. In a 40
    > million row table 100,000 for max locks works fine I our environment.
    I
    > believe MAX LOCKS should be a number relative to the size of the table.
    > Another note;
    > This is what I found about “With UR”, to me this is why it is working
    in
    > our environment.
    > Uncommitted Read: The Uncommitted Read (UR) isolation level, also
    > known
    as
    > "dirty read," is the lowest level of isolation supported by DB2. It
    > can
    be
    > used to access uncommitted data changes of other applications. For
    example,
    > an application using the Uncommitted Read isolation level will return
    all
    > of the matching rows for the query, even if that data is in the
    > process
    of
    > being modified and may not be committed to the database. You need to
    > be aware that if you are using Uncommitted Read, two identical queries
    > may
    get
    > different results, even if they are issued within a unit of work,
    > since other concurrent applications can change or modify those rows
    > that the first query retrieves.
    > Uncommitted Read transactions will hold very few locks. Thus they are
    not
    > likely to wait for other transactions to release locks. If you are
    > accessing read-only tables or it is acceptable for the application to
    > retrieve uncommitted data updated by another application, use this
    > isolation level, because it is most preferable in terms of performance.
    >
    >
    >
    >
    > "Leadership and learning are indispensable to each other." - John
    > Fitzgerald Kennedy
    >
    > 360 905-7343 Jan 31 new number is 360 905-7943
    > 360 448-8623 (c)
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 4:05 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Let me (re)-understand the problem. Each concurrent task retrieves a
    row
    > to begin processing and attempts to lock it down by issuing an update
    > statement. If this is true WITH UR may only shift the problem to the
    > update and cause excessive waits rather than timeouts. Using WITH UR
    both
    > competing tasks will still attempt to do the same piece of work. I
    believe
    > SKIP LOCKED DATA offers the best chance for success here since once
    > TASK
    A
    > issues its update, TASK B will not attempt to read the row.
    >
    > However, if the situation is more complex than my simple
    > understanding,
    of
    > course there could be other issues.
    >
    > From: Reins, Dee D [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 5:06 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > We have data sharing, and had problems with locking. We carefully
    > added row level locking to the tables we needed to, and that resolved
    > our
    issue.
    > Row level locking should not be a problem in a single DB2 environment.

    > I suggest picking a table with the most locks, make it row level
    locking.
    > The next table will bubble to the top, repeat the process. Doing this
    > with “WITH UR”, should help a lot.
    >
    >
    >
    >
    >
    > "Leadership and learning are indispensable to each other." - John
    > Fitzgerald Kennedy
    >
    > 360 905-7343 Jan 31 new number is 360 905-7943
    > 360 448-8623 (c)
    >
    > From: Campbell, Wayne [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 1:42 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Remember to use “WITH UR” only if it’s appropriate for your needs.
    > Are you seeing any lock escalations? What is your commit logic like?
    > Do
    you
    > commit after select statement? This got us a little while back, the
    > developers didn’t realize that selects took locks and left them hanging.
    > Wayne Campbell
    > DB2 DBA
    > Administrative Office of the Courts
    > (360) 705-5268
    > Email:
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 10:33 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > More than 500 within a day -- never heard of that many deadlocks not
    even
    > within a month period. Is there something fundamentally wrong -- you
    bet!!
    > For a starter, i would suggest you update all your Ad hoc READs to
    > use
    "
    > ...WITH UR".
    > Next increase your zparm timeout parm Take advantage of RCT parm
    > ROLBE=NO. And finally take advantage of some of the lock avoidance
    parms
    > available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    your
    > bind parms look good.
    >
    > If you're still getting more than 20 a day -- I'll suggest its time
    > you create anther SUBSYS. Some applications just can't coexist.
    >
    > Thx and best,
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > wrote:
    > In this case, the update does not move the row to another partition.
    >
    > From: Walter Janißen [mailto:[login to unmask email]
    > Sent: January 23, 2013 10:27 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > But that is no longer a problem as in some earlier DB2 versions.
    >
    > Mit freundlichen Grüßen
    > Walter Janißen
    >
    > ITERGO Informationstechnologie GmbH
    > Anwendungsentwicklung
    > Technische Anwendungsarchitektur
    > Victoriaplatz 2
    > D-40198 Düsseldorf
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > ITERGO Informationstechnologie GmbH
    > Vorsitzender des Aufsichtsrats: Christian Diedrich
    > Geschäftsführung: Dr. Bettina Anders (Vorsitzende), Lothar Engelke,
    > Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    Regauer
    > Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >
    >
    > ________________________________
    > Von: Basivi Inaganti [mailto:[login to unmask email]
    > Gesendet: Mittwoch, 23. Januar 2013 16:19
    > An: [login to unmask email]<mailto:[login to unmask email]>
    > Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9] Andre,
    >
    > By any chance, is the update causing the rows to move to new partition?
    >
    > Thanks,
    > ______________________________________________________________________
    > [cid:[login to unmask email]
    >
    >
    >
    > From: "LeBlanc,André-ITB"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > Date: 01/23/2013 10:01 AM
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    my
    > [DB2 z/OS v9]
    > ________________________________
    >
    >
    >
    > You are correct Charles, these are CICS COBOL packages.
    > Here are the bind parameters:
    >
    > RELEASE ( COMMIT ) ISOLATION ( CS )
    > VALIDATE ( BIND ) EXPLAIN ( YES )
    > DYNAMICRULES ( )
    > DEGREE ( 1 ) CURRENTDATA ( NO )
    > KEEPDYNAMIC ( NO )
    > DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    > PATHDEFAULT
    > IMMEDWRITE ( NO ) ENCODING ( 37 )
    >
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:48 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > What level of db2 concurrency are you using? My guess, these are
    > static
    > db2 packages -- please advise because I couldn't find these pertinent
    info
    > in your problem description.
    > Best and Thx !!!
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 8:30 AM, David Simpson
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > It’s only supposed to skip incompatible locks (I think)… Test Test
    > Test…
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:30 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > I thought of that, but since it can get the S lock, would it skip the
    row
    > in the fetch?
    >
    > Thanks,
    > André LeBlanc
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:28 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    DATA
    > clause on your cursor select in DB2 9. This feature was intended for
    > a situation like yours. If the cursor encounters a locked row it will
    skip
    > it without waiting. I would certainly test this heavily in your
    > environment to see if it’s appropriate.
    >
    >
    ______________________________________________________________________________
    > David Simpson | Senior Technical Advisor | Themis Education
    > 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    > [login to unmask email]<mailto:[login to unmask email]>
    > For more information about Themis, visit
    > www.themisinc.comhttp://www.themisinc.com
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:23 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it
    > caught my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks
    > may appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to
    > 8000 deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    > -----End Original Message-----
    >
    > -----End Original Message-----

    -----End Original Message-----

    Andre LeBlanc
    [CBSA]
    I really appreciate all your suggestions. We're looking into SKIP LOCKED DATA, with UR, and rewriting the Cursor logic.
    I'm hoping a rewrite will make the code more efficient and remove the need for either of the previous options.

    Thanks!
    Andre LeBlanc

    -----Original Message-----
    From: David Simpson [mailto:[login to unmask email]
    Sent: January 24, 2013 4:35 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Also true if SKIP LOCKED DATA is employed although I think SKIP LOCKED DATA will result in less contention than UR in this case since it will skip over any items that have already been UPDATEd by another process and won't even see them. UR will see them and then fail later.

    -----Original Message-----
    From: Daniel Luksetich [mailto:[login to unmask email]
    Sent: Thursday, January 24, 2013 3:25 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    In this particular situation the use of UR will have to go hand in hand with an optimistic locking strategy (i.e. checking timestamps on updates).
    Cheers,
    Dan

    On Thu, 24 Jan 2013 18:18:17 +0000, "Reins, Dee D"
    <[login to unmask email]> wrote:
    > We have increased MAX LOCKS on table spaces where appropriate. In a 40
    > million row table 100,000 for max locks works fine I our environment.
    I
    > believe MAX LOCKS should be a number relative to the size of the table.
    > Another note;
    > This is what I found about “With UR”, to me this is why it is working
    in
    > our environment.
    > Uncommitted Read: The Uncommitted Read (UR) isolation level, also
    > known
    as
    > "dirty read," is the lowest level of isolation supported by DB2. It
    > can
    be
    > used to access uncommitted data changes of other applications. For
    example,
    > an application using the Uncommitted Read isolation level will return
    all
    > of the matching rows for the query, even if that data is in the
    > process
    of
    > being modified and may not be committed to the database. You need to
    > be aware that if you are using Uncommitted Read, two identical queries
    > may
    get
    > different results, even if they are issued within a unit of work,
    > since other concurrent applications can change or modify those rows
    > that the first query retrieves.
    > Uncommitted Read transactions will hold very few locks. Thus they are
    not
    > likely to wait for other transactions to release locks. If you are
    > accessing read-only tables or it is acceptable for the application to
    > retrieve uncommitted data updated by another application, use this
    > isolation level, because it is most preferable in terms of performance.
    >
    >
    >
    >
    > "Leadership and learning are indispensable to each other." - John
    > Fitzgerald Kennedy
    >
    > 360 905-7343 Jan 31 new number is 360 905-7943
    > 360 448-8623 (c)
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 4:05 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Let me (re)-understand the problem. Each concurrent task retrieves a
    row
    > to begin processing and attempts to lock it down by issuing an update
    > statement. If this is true WITH UR may only shift the problem to the
    > update and cause excessive waits rather than timeouts. Using WITH UR
    both
    > competing tasks will still attempt to do the same piece of work. I
    believe
    > SKIP LOCKED DATA offers the best chance for success here since once
    > TASK
    A
    > issues its update, TASK B will not attempt to read the row.
    >
    > However, if the situation is more complex than my simple
    > understanding,
    of
    > course there could be other issues.
    >
    > From: Reins, Dee D [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 5:06 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > We have data sharing, and had problems with locking. We carefully
    > added row level locking to the tables we needed to, and that resolved
    > our
    issue.
    > Row level locking should not be a problem in a single DB2 environment.

    > I suggest picking a table with the most locks, make it row level
    locking.
    > The next table will bubble to the top, repeat the process. Doing this
    > with “WITH UR”, should help a lot.
    >
    >
    >
    >
    >
    > "Leadership and learning are indispensable to each other." - John
    > Fitzgerald Kennedy
    >
    > 360 905-7343 Jan 31 new number is 360 905-7943
    > 360 448-8623 (c)
    >
    > From: Campbell, Wayne [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 1:42 PM
    > To: '[login to unmask email]'
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Remember to use “WITH UR” only if it’s appropriate for your needs.
    > Are you seeing any lock escalations? What is your commit logic like?
    > Do
    you
    > commit after select statement? This got us a little while back, the
    > developers didn’t realize that selects took locks and left them hanging.
    > Wayne Campbell
    > DB2 DBA
    > Administrative Office of the Courts
    > (360) 705-5268
    > Email:
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 10:33 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > More than 500 within a day -- never heard of that many deadlocks not
    even
    > within a month period. Is there something fundamentally wrong -- you
    bet!!
    > For a starter, i would suggest you update all your Ad hoc READs to
    > use
    "
    > ...WITH UR".
    > Next increase your zparm timeout parm Take advantage of RCT parm
    > ROLBE=NO. And finally take advantage of some of the lock avoidance
    parms
    > available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    your
    > bind parms look good.
    >
    > If you're still getting more than 20 a day -- I'll suggest its time
    > you create anther SUBSYS. Some applications just can't coexist.
    >
    > Thx and best,
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > wrote:
    > In this case, the update does not move the row to another partition.
    >
    > From: Walter Janißen [mailto:[login to unmask email]
    > Sent: January 23, 2013 10:27 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > But that is no longer a problem as in some earlier DB2 versions.
    >
    > Mit freundlichen Grüßen
    > Walter Janißen
    >
    > ITERGO Informationstechnologie GmbH
    > Anwendungsentwicklung
    > Technische Anwendungsarchitektur
    > Victoriaplatz 2
    > D-40198 Düsseldorf
    > [login to unmask email]<mailto:[login to unmask email]>
    >
    > ITERGO Informationstechnologie GmbH
    > Vorsitzender des Aufsichtsrats: Christian Diedrich
    > Geschäftsführung: Dr. Bettina Anders (Vorsitzende), Lothar Engelke,
    > Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    Regauer
    > Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >
    >
    > ________________________________
    > Von: Basivi Inaganti [mailto:[login to unmask email]
    > Gesendet: Mittwoch, 23. Januar 2013 16:19
    > An: [login to unmask email]<mailto:[login to unmask email]>
    > Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9] Andre,
    >
    > By any chance, is the update causing the rows to move to new partition?
    >
    > Thanks,
    > ______________________________________________________________________
    > [cid:[login to unmask email]
    >
    >
    >
    > From: "LeBlanc,André-ITB"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    > <[login to unmask email]<mailto:[login to unmask email]>>
    > Date: 01/23/2013 10:01 AM
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    my
    > [DB2 z/OS v9]
    > ________________________________
    >
    >
    >
    > You are correct Charles, these are CICS COBOL packages.
    > Here are the bind parameters:
    >
    > RELEASE ( COMMIT ) ISOLATION ( CS )
    > VALIDATE ( BIND ) EXPLAIN ( YES )
    > DYNAMICRULES ( )
    > DEGREE ( 1 ) CURRENTDATA ( NO )
    > KEEPDYNAMIC ( NO )
    > DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    > PATHDEFAULT
    > IMMEDWRITE ( NO ) ENCODING ( 37 )
    >
    >
    > From: Charles Browm [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:48 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Andre
    > What level of db2 concurrency are you using? My guess, these are
    > static
    > db2 packages -- please advise because I couldn't find these pertinent
    info
    > in your problem description.
    > Best and Thx !!!
    >
    > Sent from my iPhone
    >
    > On Jan 23, 2013, at 8:30 AM, David Simpson
    > <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    > It’s only supposed to skip incompatible locks (I think)… Test Test
    > Test…
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:30 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > I thought of that, but since it can get the S lock, would it skip the
    row
    > in the fetch?
    >
    > Thanks,
    > André LeBlanc
    >
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: January 23, 2013 9:28 AM
    > To: [login to unmask email]<mailto:[login to unmask email]>
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    DATA
    > clause on your cursor select in DB2 9. This feature was intended for
    > a situation like yours. If the cursor encounters a locked row it will
    skip
    > it without waiting. I would certainly test this heavily in your
    > environment to see if it’s appropriate.
    >
    >
    ______________________________________________________________________________
    > David Simpson | Senior Technical Advisor | Themis Education
    > 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    > [login to unmask email]<mailto:[login to unmask email]>
    > For more information about Themis, visit
    > www.themisinc.comhttp://www.themisinc.com
    >
    > From: LeBlanc, André-ITB [mailto:[login to unmask email]
    > Sent: Wednesday, January 23, 2013 8:23 AM
    > To: '[login to unmask email]<mailto:[login to unmask email]>'
    > Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it
    > caught my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks
    > may appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to
    > 8000 deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    >
    > -----End Original Message-----
    > -----End Original Message-----
    >
    > -----End Original Message-----

    -----End Original Message-----


    -----End Original Message-----

    Daniel Luksetich
    [DanL Database Consulting]
    Make sure to tell us what happens!
    Dan
    On Fri, 25 Jan 2013 14:29:20 +0000, LeBlanc, André-ITB
    <[login to unmask email]> wrote:
    > I really appreciate all your suggestions. We're looking into SKIP
    LOCKED
    > DATA, with UR, and rewriting the Cursor logic.
    > I'm hoping a rewrite will make the code more efficient and remove the
    need
    > for either of the previous options.
    >
    > Thanks!
    > Andre LeBlanc
    >
    > -----Original Message-----
    > From: David Simpson [mailto:[login to unmask email]
    > Sent: January 24, 2013 4:35 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > Also true if SKIP LOCKED DATA is employed although I think SKIP LOCKED
    > DATA will result in less contention than UR in this case since it will
    skip
    > over any items that have already been UPDATEd by another process and
    won't
    > even see them. UR will see them and then fail later.
    >
    > -----Original Message-----
    > From: Daniel Luksetich [mailto:[login to unmask email]
    > Sent: Thursday, January 24, 2013 3:25 PM
    > To: [login to unmask email]
    > Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    > [DB2 z/OS v9]
    >
    > In this particular situation the use of UR will have to go hand in hand
    > with an optimistic locking strategy (i.e. checking timestamps on
    updates).
    > Cheers,
    > Dan
    >
    > On Thu, 24 Jan 2013 18:18:17 +0000, "Reins, Dee D"
    > <[login to unmask email]> wrote:
    >> We have increased MAX LOCKS on table spaces where appropriate. In a
    40
    >> million row table 100,000 for max locks works fine I our environment.
    > I
    >> believe MAX LOCKS should be a number relative to the size of the table.
    >> Another note;
    >> This is what I found about “With UR”, to me this is why it is working
    > in
    >> our environment.
    >> Uncommitted Read: The Uncommitted Read (UR) isolation level, also
    >> known
    > as
    >> "dirty read," is the lowest level of isolation supported by DB2. It
    >> can
    > be
    >> used to access uncommitted data changes of other applications. For
    > example,
    >> an application using the Uncommitted Read isolation level will return
    > all
    >> of the matching rows for the query, even if that data is in the
    >> process
    > of
    >> being modified and may not be committed to the database. You need to
    >> be aware that if you are using Uncommitted Read, two identical queries
    >> may
    > get
    >> different results, even if they are issued within a unit of work,
    >> since other concurrent applications can change or modify those rows
    >> that the first query retrieves.
    >> Uncommitted Read transactions will hold very few locks. Thus they are
    > not
    >> likely to wait for other transactions to release locks. If you are
    >> accessing read-only tables or it is acceptable for the application to
    >> retrieve uncommitted data updated by another application, use this
    >> isolation level, because it is most preferable in terms of performance.
    >>
    >>
    >>
    >>
    >> "Leadership and learning are indispensable to each other." - John
    >> Fitzgerald Kennedy
    >>
    >> 360 905-7343 Jan 31 new number is 360 905-7943
    >> 360 448-8623 (c)
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 4:05 PM
    >> To: [login to unmask email]
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Let me (re)-understand the problem. Each concurrent task retrieves a
    > row
    >> to begin processing and attempts to lock it down by issuing an update
    >> statement. If this is true WITH UR may only shift the problem to the
    >> update and cause excessive waits rather than timeouts. Using WITH UR
    > both
    >> competing tasks will still attempt to do the same piece of work. I
    > believe
    >> SKIP LOCKED DATA offers the best chance for success here since once
    >> TASK
    > A
    >> issues its update, TASK B will not attempt to read the row.
    >>
    >> However, if the situation is more complex than my simple
    >> understanding,
    > of
    >> course there could be other issues.
    >>
    >> From: Reins, Dee D [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 5:06 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> We have data sharing, and had problems with locking. We carefully
    >> added row level locking to the tables we needed to, and that resolved
    >> our
    > issue.
    >> Row level locking should not be a problem in a single DB2 environment.

    >
    >> I suggest picking a table with the most locks, make it row level
    > locking.
    >> The next table will bubble to the top, repeat the process. Doing this
    >> with “WITH UR”, should help a lot.
    >>
    >>
    >>
    >>
    >>
    >> "Leadership and learning are indispensable to each other." - John
    >> Fitzgerald Kennedy
    >>
    >> 360 905-7343 Jan 31 new number is 360 905-7943
    >> 360 448-8623 (c)
    >>
    >> From: Campbell, Wayne [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 1:42 PM
    >> To: '[login to unmask email]'
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Remember to use “WITH UR” only if it’s appropriate for your needs.
    >> Are you seeing any lock escalations? What is your commit logic like?

    >> Do
    > you
    >> commit after select statement? This got us a little while back, the
    >> developers didn’t realize that selects took locks and left them
    hanging.
    >> Wayne Campbell
    >> DB2 DBA
    >> Administrative Office of the Courts
    >> (360) 705-5268
    >> Email:
    >> [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 10:33 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> More than 500 within a day -- never heard of that many deadlocks not
    > even
    >> within a month period. Is there something fundamentally wrong -- you
    > bet!!
    >> For a starter, i would suggest you update all your Ad hoc READs to
    >> use
    > "
    >> ...WITH UR".
    >> Next increase your zparm timeout parm Take advantage of RCT parm
    >> ROLBE=NO. And finally take advantage of some of the lock avoidance
    > parms
    >> available in v9 such as SKIPUNCI -- someone has already mentioned. BTW
    > your
    >> bind parms look good.
    >>
    >> If you're still getting more than 20 a day -- I'll suggest its time
    >> you create anther SUBSYS. Some applications just can't coexist.
    >>
    >> Thx and best,
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 9:30 AM, LeBlanc, André-ITB
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> wrote:
    >> In this case, the update does not move the row to another partition.
    >>
    >> From: Walter Janißen [mailto:[login to unmask email]
    >> Sent: January 23, 2013 10:27 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - AW: RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> But that is no longer a problem as in some earlier DB2 versions.
    >>
    >> Mit freundlichen Grüßen
    >> Walter Janißen
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Anwendungsentwicklung
    >> Technische Anwendungsarchitektur
    >> Victoriaplatz 2
    >> D-40198 Düsseldorf
    >> [login to unmask email]<mailto:[login to unmask email]>
    >>
    >> ITERGO Informationstechnologie GmbH
    >> Vorsitzender des Aufsichtsrats: Christian Diedrich
    >> Geschäftsführung: Dr. Bettina Anders (Vorsitzende), Lothar Engelke,
    >> Ina Kirchhof, Dr. Christian Nymphius, Dr. Michael
    > Regauer
    >> Sitz: Düsseldorf, Handelsregister: Amtsgericht Düsseldorf HRB 37996
    >>
    >>
    >> ________________________________
    >> Von: Basivi Inaganti [mailto:[login to unmask email]
    >> Gesendet: Mittwoch, 23. Januar 2013 16:19
    >> An: [login to unmask email]<mailto:[login to unmask email]>
    >> Betreff: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9] Andre,
    >>
    >> By any chance, is the update causing the rows to move to new partition?
    >>
    >> Thanks,
    >> ______________________________________________________________________
    >> [cid:[login to unmask email]
    >>
    >>
    >>
    >> From: "LeBlanc,André-ITB"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> To: "'[login to unmask email]<mailto:[login to unmask email]>'"
    >> <[login to unmask email]<mailto:[login to unmask email]>>
    >> Date: 01/23/2013 10:01 AM
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh
    > my
    >> [DB2 z/OS v9]
    >> ________________________________
    >>
    >>
    >>
    >> You are correct Charles, these are CICS COBOL packages.
    >> Here are the bind parameters:
    >>
    >> RELEASE ( COMMIT ) ISOLATION ( CS )
    >> VALIDATE ( BIND ) EXPLAIN ( YES )
    >> DYNAMICRULES ( )
    >> DEGREE ( 1 ) CURRENTDATA ( NO )
    >> KEEPDYNAMIC ( NO )
    >> DBPROTOCOL ( DRDA ) NOREOPT ( VARS )
    >> PATHDEFAULT
    >> IMMEDWRITE ( NO ) ENCODING ( 37 )
    >>
    >>
    >> From: Charles Browm [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:48 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Andre
    >> What level of db2 concurrency are you using? My guess, these are
    >> static
    >> db2 packages -- please advise because I couldn't find these pertinent
    > info
    >> in your problem description.
    >> Best and Thx !!!
    >>
    >> Sent from my iPhone
    >>
    >> On Jan 23, 2013, at 8:30 AM, David Simpson
    >> <[login to unmask email]<mailto:[login to unmask email]>> wrote:
    >> It’s only supposed to skip incompatible locks (I think)… Test Test
    >> Test…
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:30 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> I thought of that, but since it can get the S lock, would it skip the
    > row
    >> in the fetch?
    >>
    >> Thanks,
    >> André LeBlanc
    >>
    >> From: David Simpson [mailto:[login to unmask email]
    >> Sent: January 23, 2013 9:28 AM
    >> To: [login to unmask email]<mailto:[login to unmask email]>
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> While I’m not sure it’s a good fit you could consider the SKIP LOCKED
    > DATA
    >> clause on your cursor select in DB2 9. This feature was intended for
    >> a situation like yours. If the cursor encounters a locked row it will
    > skip
    >> it without waiting. I would certainly test this heavily in your
    >> environment to see if it’s appropriate.
    >>
    >>
    >
    ______________________________________________________________________________
    >> David Simpson | Senior Technical Advisor | Themis Education
    >> 123 South Avenue E, Westfield NJ, 07090 | phone 908-233-8900 |
    >> [login to unmask email]<mailto:[login to unmask email]>
    >> For more information about Themis, visit
    >> www.themisinc.comhttp://www.themisinc.com
    >>
    >> From: LeBlanc, André-ITB [mailto:[login to unmask email]
    >> Sent: Wednesday, January 23, 2013 8:23 AM
    >> To: '[login to unmask email]<mailto:[login to unmask email]>'
    >> Subject: [DB2-L] - RE: Cursors, Partitions, and Deadlocks... Oh my
    >> [DB2 z/OS v9]
    >>
    >> Esteemed Listers, I can really use your help.
    >>
    >> When we started getting 800 deadlocks a day on one tablespace, it
    >> caught my attention.
    >> And then it climbed to 8000 deadlocks per day!
    >>
    >> Here’s what I found:
    >> - Our application needs to process data by a given Timestamp,
    >> assigned based on priority.
    >> - Some data has up to 4 hours to be processed, others must be
    >> processed within 2 minutes
    >> - Some rows may be related to other rows (A few or a few
    >> thousand) in a separate relationship table
    >> - All related data must be processed in the sequence it was
    >> captured.
    >> - Tablespace has 120 parts, partitioned and clustered by
    >> day/hour, with ROW level locking.
    >> - At any given time, 1 partition is very busy, 2 partitions
    are
    >> somewhat busy, and all other partitions are empty.
    >> - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    >> HOLD so that data inserted with earlier timestamp is picked up
    > immediately
    >> - Cursor has no ORDER BY. Access to earliest timestamp is
    > based
    >> on DB2 accessing earlier partitions first
    >> - Multiple threads compete to process data in this table
    >> - When a thread finds a candidate row, it claims it by issuing
    > a
    >> separate UPDATE outside of cursor. (not where current of cursor)
    >>
    >> The transaction involved in the deadlocks is as follows:
    >> - Open Cursor
    >> - Fetch1
    >> - Select1 (if earlier related rows exist, fetch next row)
    >> - Fetch2
    >> - Select2
    >> - …
    >> - Fetch44
    >> - Select44 (rc=+100) no earlier related row found so thread
    > will
    >> try to claim row
    >> - Update row using values from Fetch44 (will get +100 if row
    >> already updated by another thread)
    >> - Commit
    >>
    >> Deadlock occurs because multiple threads are all executing this same
    >> transaction at the same time.
    >> 1- The first thread to get to the update gets U lock on row1.
    >> 2- Thread2 grab a S lock on row1 during Fetch44.
    >> 3- Thread1 wants to change its U lock to X lock ?C waits on
    Thread2
    > S
    >> lock
    >> 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    > Thread1
    >> U lock
    >>
    >> I have an index that gives the Cursor INDEX ONLY access, so I actually
    >> don’t see any row locks for most Fetches. It appears that row locks
    >> may appear when another thread attempts an UPDATE.
    >>
    >> We see 800 deadlocks a day when running 4 competing threads.
    >> By going to 6 threads they doubled the throughput, and increased to
    >> 8000 deadlocks a day.
    >>
    >> Everything runs in milliseconds since these pages are always in Buffer
    >> Pool so overall response time is within the SLA.
    >> There is retry logic, so the end user is not that concerned about the
    >> deadlocks.
    >>
    >> I’m not sure what my next step should be, so any suggestions you may
    > have
    >> are greatly appreciated.
    >>
    >> Cheers!
    >> Andr?? LeBlanc
    >>
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >> -----End Original Message-----
    >>
    >> -----End Original Message-----
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----
    >
    >
    > -----End Original Message-----
    Lockwood Lyon
    [Fifth Third Bankcorp]
    Andre,

    I'm a bit surprised that no one suggested the following (apologies if I missed it.)

    You are using CICS transactions, and you report that they are "finished within milliseconds". Have you considered adding an ORDER BY to your cursor, adding WITH UPDATE OF, and doing your update with a WHERE CURRENT OF ... ?

    This would effectively serialize your transactions, since you are already doing row-level locking and accessing fields by timestampr. Serialized access to the rows should eliminate most of your deadlock issues.

    Of course, your original issue description was limited to the deadlock symptom(s). For example, if you were also experiencing relatively long wait times for VP pages, that would be another discussion.

    Last, I didn't see a lot of specifics on other applications that access this data, whether it must be available 24x7, how (or if) you schedule Reorgs, etc. I also didn't see the mechanism for Load/Insert of rows to populate this table. These things will also affect possible solutions.

    Good luck!

    Lock Lyon
    Fifth Third Bancorp


    -----Original Message-----
    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Friday, January 25, 2013 9:29 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I really appreciate all your suggestions. We're looking into SKIP LOCKED DATA, with UR, and rewriting the Cursor logic.
    I'm hoping a rewrite will make the code more efficient and remove the need for either of the previous options.

    Thanks!
    Andre LeBlanc

    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it
    > caught my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks
    > may appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to
    > 8000 deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >


    This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.
    Andre LeBlanc
    [CBSA]
    I'm glad you brought it up Lyon. I'm also concerned that the ordering of the rows is based on our partitioning scheme, which is why I'd like to have an explicit ORDER BY. But the coder is worried the cursor would materialize and become static, which would prevent them from seeing any high priority rows inserted after the OPEN CURSOR. However, when I tested ORDER BY, the cursor actually used IX only to access and I did not see any result set materialization.

    Unless I'm mistaken, the ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions for the order?

    Cheers!
    Andre
    -----Original Message-----
    From: Lyon, Lockwood [mailto:[login to unmask email]
    Sent: January 25, 2013 10:17 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    I'm a bit surprised that no one suggested the following (apologies if I missed it.)

    You are using CICS transactions, and you report that they are "finished within milliseconds". Have you considered adding an ORDER BY to your cursor, adding WITH UPDATE OF, and doing your update with a WHERE CURRENT OF ... ?

    This would effectively serialize your transactions, since you are already doing row-level locking and accessing fields by timestampr. Serialized access to the rows should eliminate most of your deadlock issues.

    Of course, your original issue description was limited to the deadlock symptom(s). For example, if you were also experiencing relatively long wait times for VP pages, that would be another discussion.

    Last, I didn't see a lot of specifics on other applications that access this data, whether it must be available 24x7, how (or if) you schedule Reorgs, etc. I also didn't see the mechanism for Load/Insert of rows to populate this table. These things will also affect possible solutions.

    Good luck!

    Lock Lyon
    Fifth Third Bancorp


    -----Original Message-----
    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Friday, January 25, 2013 9:29 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I really appreciate all your suggestions. We're looking into SKIP LOCKED DATA, with UR, and rewriting the Cursor logic.
    I'm hoping a rewrite will make the code more efficient and remove the need for either of the previous options.

    Thanks!
    Andre LeBlanc

    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it
    > caught my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks
    > may appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to
    > 8000 deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >


    This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.

    -----End Original Message-----

    Lockwood Lyon
    [Fifth Third Bankcorp]
    Andre,

    The issues of Order By, result set materialization, cursor behavior available with asensitive / sensitive / insensitive, dynamic / static, with / without hold, Bind options like Isolation Level, table lock options, commit frequency, STDSQL and NOFOR compile options, and so forth, generate lots of combinations. Honestly, I haven't dealt with all of the combinations in any detail to offer specific recommendations or guidelines.

    I am not aware of a V9 rule that prohibits using ORDER BY and FOR UPDATE OF together. Perhaps this is because you have chosen a sensitive dynamic cursor.

    I do not believe you can rely on your partitioning scheme to maintain rows on physical timestamp order. When DB2 inserts a row on a page it will attempt to add it based on the clustering index; however, if no space is available on that page DB2 will insert wherever it can. In my experience, you cannot rely on the table/index DDL to guarantee physical row placement within pages, or retrieval with a SELECT which does not specify an ORDER BY. While there have been several ways described to ensure that rows are only Inserted to the 'end' of a table, I am also certain that IBM's response has always been "No ORDER BY, no promise to return an ordered result set".

    If the appearance of 'high-priority rows while the cursor is open' is and a critical part of the application, then I suspect that we simply don't have enough information about the application requirements to give you a solid DB2 "database design + application design" answer. In general, the business rules will drive the design. Specifically, we would need to know a lot more about what constitutes 'rows of related data' (transactions?), how they were related (and reasons for the design of the separate relationship table), how they are presented to the DBMS (row inserts from a distributed application? Data arriving on an MQSeries queue?), and other information that I suspect is confidential.

    By the way, congratulations on being assigned to support a complex, mission-critical application! Get this right, and you've got a for-sure IDUG presentation!

    In my opinion, this particular application and its implementation require more than the e-mails and tips that this forum can give you. Despite your need for a quick answer, I believe the best option is to engage some experienced help. Submit a question to IBM, engage an expert consultant. Perhaps some members of this forum who have supported similar systems can give you advice.

    Good luck.

    Lock Lyon




    -----Original Message-----
    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Friday, January 25, 2013 11:46 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I'm glad you brought it up Lyon. I'm also concerned that the ordering of the rows is based on our partitioning scheme, which is why I'd like to have an explicit ORDER BY. But the coder is worried the cursor would materialize and become static, which would prevent them from seeing any high priority rows inserted after the OPEN CURSOR. However, when I tested ORDER BY, the cursor actually used IX only to access and I did not see any result set materialization.

    Unless I'm mistaken, the ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions for the order?

    Cheers!
    Andre
    -----Original Message-----
    From: Lyon, Lockwood [mailto:[login to unmask email]
    Sent: January 25, 2013 10:17 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    I'm a bit surprised that no one suggested the following (apologies if I missed it.)

    You are using CICS transactions, and you report that they are "finished within milliseconds". Have you considered adding an ORDER BY to your cursor, adding WITH UPDATE OF, and doing your update with a WHERE CURRENT OF ... ?

    This would effectively serialize your transactions, since you are already doing row-level locking and accessing fields by timestampr. Serialized access to the rows should eliminate most of your deadlock issues.

    Of course, your original issue description was limited to the deadlock symptom(s). For example, if you were also experiencing relatively long wait times for VP pages, that would be another discussion.

    Last, I didn't see a lot of specifics on other applications that access this data, whether it must be available 24x7, how (or if) you schedule Reorgs, etc. I also didn't see the mechanism for Load/Insert of rows to populate this table. These things will also affect possible solutions.

    Good luck!

    Lock Lyon
    Fifth Third Bancorp


    -----Original Message-----
    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Friday, January 25, 2013 9:29 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I really appreciate all your suggestions. We're looking into SKIP LOCKED DATA, with UR, and rewriting the Cursor logic.
    I'm hoping a rewrite will make the code more efficient and remove the need for either of the previous options.

    Thanks!
    Andre LeBlanc

    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it
    > caught my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks
    > may appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to
    > 8000 deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >


    This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.
    Peter Vanroose
    [ABIS Training & Consulting]

    A sensitive static scrollable cursor always materialises, and allows a combined ORDER BY and FOR UPDATE.

    A dynamic scrollable cursor never materializes, but does not allow a combined ORDER BY and FOR UPDATE.

    (And, of course, a non-sensitive static scroll cursor does not allow FOR UPDATE; and finally, a non-scrollable cursor allows any but not both of the options.)

    In Reply to Andre LeBlanc:

    [...] Unless I'm mistaken, the ORDER BY would preclude FOR UPDATE OF [...]


    --      Peter Vanroose
            ABIS Training & Consulting,
            Leuven, Belgium.
            http://www.abis.be/

    Andre LeBlanc
    [CBSA]
    Lyon, I couldn't agree with you more. This is a very complex, mission-critical HA application, and I can't provide all the information or get all the answers by email. But I did get a lot!

    Being the lone DBA trying to come up with an alternative to this complex application logic, I need to know that I have all my DB2 facts straight. When I mentioned that no ORDER BY meant no guaranteed order in result set, the application did not think this was a valid concern. Validating this part of the design was one of my objectives, so thank you for that.

    The good news is that I was able to merge the CURSOR and the SELECT into a singled ORDERED cursor that performs the existence check using a NOT EXISTS correlated sub-query. If the testing confirms that the new cursor always correctly returns the earliest related row, then it would only require a single fetch, and no subsequent SELECT, before performing the UPDATE.

    If the SQL for new cursor is approved, I don't know if we still need a CURSOR at all.
    Would it be less overhead to perform a single SELECT - FETCH FIRST instead of using a cursor?
    (If this is my only conundrum now, I'm very happy!) :D

    Finally, preliminary tests of UPDATE SKIP LOCK seem very promising. It might not even be required if we're able to completely redo the transaction with the new cursor SQL.

    So things are looking up, and I'm cautiously optimistic that we may have a working solution within reach.

    Thank you all so much!
    Andre

    -----Original Message-----
    From: Lyon, Lockwood [mailto:[login to unmask email]
    Sent: January 25, 2013 1:57 PM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    The issues of Order By, result set materialization, cursor behavior available with asensitive / sensitive / insensitive, dynamic / static, with / without hold, Bind options like Isolation Level, table lock options, commit frequency, STDSQL and NOFOR compile options, and so forth, generate lots of combinations. Honestly, I haven't dealt with all of the combinations in any detail to offer specific recommendations or guidelines.

    I am not aware of a V9 rule that prohibits using ORDER BY and FOR UPDATE OF together. Perhaps this is because you have chosen a sensitive dynamic cursor.

    I do not believe you can rely on your partitioning scheme to maintain rows on physical timestamp order. When DB2 inserts a row on a page it will attempt to add it based on the clustering index; however, if no space is available on that page DB2 will insert wherever it can. In my experience, you cannot rely on the table/index DDL to guarantee physical row placement within pages, or retrieval with a SELECT which does not specify an ORDER BY. While there have been several ways described to ensure that rows are only Inserted to the 'end' of a table, I am also certain that IBM's response has always been "No ORDER BY, no promise to return an ordered result set".

    If the appearance of 'high-priority rows while the cursor is open' is and a critical part of the application, then I suspect that we simply don't have enough information about the application requirements to give you a solid DB2 "database design + application design" answer. In general, the business rules will drive the design. Specifically, we would need to know a lot more about what constitutes 'rows of related data' (transactions?), how they were related (and reasons for the design of the separate relationship table), how they are presented to the DBMS (row inserts from a distributed application? Data arriving on an MQSeries queue?), and other information that I suspect is confidential.

    By the way, congratulations on being assigned to support a complex, mission-critical application! Get this right, and you've got a for-sure IDUG presentation!

    In my opinion, this particular application and its implementation require more than the e-mails and tips that this forum can give you. Despite your need for a quick answer, I believe the best option is to engage some experienced help. Submit a question to IBM, engage an expert consultant. Perhaps some members of this forum who have supported similar systems can give you advice.

    Good luck.

    Lock Lyon




    -----Original Message-----
    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Friday, January 25, 2013 11:46 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I'm glad you brought it up Lyon. I'm also concerned that the ordering of the rows is based on our partitioning scheme, which is why I'd like to have an explicit ORDER BY. But the coder is worried the cursor would materialize and become static, which would prevent them from seeing any high priority rows inserted after the OPEN CURSOR. However, when I tested ORDER BY, the cursor actually used IX only to access and I did not see any result set materialization.

    Unless I'm mistaken, the ORDER BY would preclude FOR UPDATE OF, but can we rely on partitions for the order?

    Cheers!
    Andre
    -----Original Message-----
    From: Lyon, Lockwood [mailto:[login to unmask email]
    Sent: January 25, 2013 10:17 AM
    To: [login to unmask email]
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    Andre,

    I'm a bit surprised that no one suggested the following (apologies if I missed it.)

    You are using CICS transactions, and you report that they are "finished within milliseconds". Have you considered adding an ORDER BY to your cursor, adding WITH UPDATE OF, and doing your update with a WHERE CURRENT OF ... ?

    This would effectively serialize your transactions, since you are already doing row-level locking and accessing fields by timestampr. Serialized access to the rows should eliminate most of your deadlock issues.

    Of course, your original issue description was limited to the deadlock symptom(s). For example, if you were also experiencing relatively long wait times for VP pages, that would be another discussion.

    Last, I didn't see a lot of specifics on other applications that access this data, whether it must be available 24x7, how (or if) you schedule Reorgs, etc. I also didn't see the mechanism for Load/Insert of rows to populate this table. These things will also affect possible solutions.

    Good luck!

    Lock Lyon
    Fifth Third Bancorp


    -----Original Message-----
    From: LeBlanc, André-ITB [mailto:[login to unmask email]
    Sent: Friday, January 25, 2013 9:29 AM
    To: '[login to unmask email]'
    Subject: [DB2-L] - RE: AW: Cursors, Partitions, and Deadlocks... Oh my [DB2 z/OS v9]

    I really appreciate all your suggestions. We're looking into SKIP LOCKED DATA, with UR, and rewriting the Cursor logic.
    I'm hoping a rewrite will make the code more efficient and remove the need for either of the previous options.

    Thanks!
    Andre LeBlanc

    >
    > Esteemed Listers, I can really use your help.
    >
    > When we started getting 800 deadlocks a day on one tablespace, it
    > caught my attention.
    > And then it climbed to 8000 deadlocks per day!
    >
    > Here’s what I found:
    > - Our application needs to process data by a given Timestamp,
    > assigned based on priority.
    > - Some data has up to 4 hours to be processed, others must be
    > processed within 2 minutes
    > - Some rows may be related to other rows (A few or a few
    > thousand) in a separate relationship table
    > - All related data must be processed in the sequence it was
    > captured.
    > - Tablespace has 120 parts, partitioned and clustered by
    > day/hour, with ROW level locking.
    > - At any given time, 1 partition is very busy, 2 partitions are
    > somewhat busy, and all other partitions are empty.
    > - Data is fetched from a SENSITIVE DYNAMIC SCROLL CURSOR WITH
    > HOLD so that data inserted with earlier timestamp is picked up
    immediately
    > - Cursor has no ORDER BY. Access to earliest timestamp is
    based
    > on DB2 accessing earlier partitions first
    > - Multiple threads compete to process data in this table
    > - When a thread finds a candidate row, it claims it by issuing
    a
    > separate UPDATE outside of cursor. (not where current of cursor)
    >
    > The transaction involved in the deadlocks is as follows:
    > - Open Cursor
    > - Fetch1
    > - Select1 (if earlier related rows exist, fetch next row)
    > - Fetch2
    > - Select2
    > - …
    > - Fetch44
    > - Select44 (rc=+100) no earlier related row found so thread
    will
    > try to claim row
    > - Update row using values from Fetch44 (will get +100 if row
    > already updated by another thread)
    > - Commit
    >
    > Deadlock occurs because multiple threads are all executing this same
    > transaction at the same time.
    > 1- The first thread to get to the update gets U lock on row1.
    > 2- Thread2 grab a S lock on row1 during Fetch44.
    > 3- Thread1 wants to change its U lock to X lock ?C waits on Thread2
    S
    > lock
    > 4- Thread2 issues UPDATE and tries to get U lock ?C waits on
    Thread1
    > U lock
    >
    > I have an index that gives the Cursor INDEX ONLY access, so I actually
    > don’t see any row locks for most Fetches. It appears that row locks
    > may appear when another thread attempts an UPDATE.
    >
    > We see 800 deadlocks a day when running 4 competing threads.
    > By going to 6 threads they doubled the throughput, and increased to
    > 8000 deadlocks a day.
    >
    > Everything runs in milliseconds since these pages are always in Buffer
    > Pool so overall response time is within the SLA.
    > There is retry logic, so the end user is not that concerned about the
    > deadlocks.
    >
    > I’m not sure what my next step should be, so any suggestions you may
    have
    > are greatly appreciated.
    >
    > Cheers!
    > Andr?? LeBlanc
    >
    >
    > -----End Original Message-----
    >


    This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.

    -----End Original Message-----

    Peter Vanroose
    [ABIS Training & Consulting]

    In Reply to Andre LeBlanc:

    Would it be less overhead to perform a single SELECT - FETCH FIRST instead of using a cursor?


    Yes -- definitely.

    --      Peter Vanroose
            ABIS Training & Consulting,
            Leuven, Belgium.
            http://www.abis.be/


    All Times America/New_York

    Copyright © 2014 IDUG. All Rights Reserved

    All material, files, logos and trademarks within this site are properties of their respective organizations.

    Terms of Service - Privacy Policy - Contact