Db2 z/OS v12 - Object Comparison Tool automation

Johan Sundborg

Db2 z/OS v12 - Object Comparison Tool automation

Hi,

Me and my DBA colleagues are currently looking into OCT (Object Comparison Tool) as a way to improve stability and simplify propagation of changes from one Db2 environment to another. I think OCT is a great tool and got good experience of it at my previous employer, however we always used OCT step by step manually.

As it stand now in my current company we got lots of Db2 environments (over 10) and it would very tedious to do OCT compare and apply for each and everyone of them as soon we for example need to add a new column.

Instead I would like to expand on one of our current home-made ISPF-panels where we can name a datafile containing DDL and chose which environments the DDL should be applied to (this setup doesn't do clever stuff like reorg, we leave that to Automation Tool), and leave the propagation and OCT usage to the panel job.

My guess is that this would either

1) require OCT to be able to be run completely through batch where you set certain parameters, such as mask-/ignore-file-name, objects to be compared, which environment are target/source, etc etc, in the batchjob.

2) Or you grab an OCT-generated compare job and modify it to a template for the panel job. When you then run the panel, which creates jobs, it adjust the template for each job, one job per chosen environment. Then the panel run the compare jobs, which creates the execution jobs, which the panel also runs (this of course require you to give OCT a certain amount of trust).

I hope I'm not making things too fuzzy by having the whole idea in my head and managing to few details. What do you think, is this doable and have someone already done it?

If someone has done something similar to this or something completely different but successful OCT automation or have any clever insight, I'm all ears!

Kind regards

Johan Sundborg, Db2 DBA

Jørn Thyssen

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Johan Sundborg)


Hi Johan,

Option 1 is available through the CM Batch feature of IBM Db2 Object Comparison Tool. I have several customers automating compares on top of that. 

I will post an example tomorrow. 

Best regards,

Jørn Thyssen

Rocket Software
77 Fourth Avenue • Waltham, MA • 02451 • USA
E: [login to unmask email] • W: www.rocketsoftware.com 

2018 IBM Champion.

Views are personal. 

Michael Hannan

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Johan Sundborg)

Johan,

I have written my own environment comparison tools in the past, using SQL Query against DB2 Catalog Tables. The last one was written just one year ago very quickly using OLAP functions just to list  the differences between many envs, e.g. could handle 10 envs all compared together, not just a compare of a pair of envs. This was just for reporting or checking, to know if logical unloads of tables could be loaded to other envs. I did not want to pay for a compare tool when I could whip up my own so quickly.

I have also previously written an SQL to generate DDL from the catalog (now quite out of date and needs a redo), so it is possible to combine the 2 things into a crude tool to generate DDL, where differences are detected, if we define which env is the base Master to be matched by the others. Not the Reorgs (although actually I would like to build my own Reorg generate tool as well - just never had time). Based on the relative lack of huge difficulties, if you should buy a tool from a 3rd party software company, it should not be that expensive in my opinion (without knowing actual cost) unless is a full change management tool.

So just doing a compare is easy, and generate DDL is not massively hard, anymore with such powerful SQL transforms these days. V8 Recursion allow Partition Limit keys in Catalog to be handled well. My original DDL generator was written well before that but did not cover XML, DPSIs, Include cols etc.

I can even supply my recent quick SQL example used to compare environments, of table columns in the Catalog tables. It was easy due to many envs all in same DB2 subsystem. If not, one would simply have to extract data into a common place using cross loader, before the compare. Obviously one problem is I understand my SQL (mostly) and others might not. Therefore I can easily extend it handle additional things when needed. The SQL is instructive though to show how OLAP functions are perfect to compare data rows from multiple sources One can give a Rank number to each different combination of a comparison key.

Yes you have to decide what types of differences matter to you. I once worked in a shop where I found the tables in different environments did not have the columns in consistent sequence. Does it matter? Not really. The COLNAMEs are more important than the COLNOs. Db2 has reordered row format these days anyway.  Obviously PRIQTY and SECQTY are things that don't need to match.  

 with cols as 
(select colno CNO
,count(*) over(partition by tbname, name) as ENVS
,dense_rank() over(partition by tbname, name
order by coltype, length, scale, nulls, default, keyseq
) as RNK1
,dense_rank() over(partition by tbname, name
order by colno
) as RNK2
,name col
,tbname TB
,tbcreator CR
,coltype TY
,length LE
,scale sc
,nulls nu
,default DF
,KEYSEQ PK
,createdts CRE
,alteredts alt
from sysibm.syscolumns
where tbcreator LIKE 'BEN_RFDB'
and TBCREATOR between 'BENA' and 'BENZ'
)
,cols2 as
(select c.*
,max(rnk1)
over (partition by TB, COL) as diff1
,max(rnk2)
over (partition by TB, COL) as diff2
from cols c
)
select *
from cols2
where diff1 > 1 or diff2 > 1
order by tb, CNO, CR
fetch first 1000 rows only
with ur;

The above example compares Table Columns only, but same principal can be used to compare other Catalog tables, or even joined Catalog tables, and small modifications can be used to convert missing values to defaults or whatever, e.g. indexes with Varchar before DB2 V8 were Padded by default. Plug in the columns that matter in the compare into the OLAP rank function. The above compare is COLNO sensitive in rnk2 and in the ORDER BY, but it could have been written to use column names as the main sort basis where COLNOs were not seen as important.

I made a slight mod to my SQL so hope did not introduce a syntax error. The Table Authids need to be changed to suit your environments.

Also need to decide which DB2 Catalog tables (and which columns) will be in the compare. Not that hard to whip up compares for Tablespaces, Partitions, Indexes, Index Partitioning, Rels/Foreignkeys, Aliases, even Views.

A 2nd set of compares can be useful for DB2 Catalog Stats relevant to the Optimizer, where Stats are migrated between envs. I have also built my own Stats migrate tools in the past, by generating DML to place Stats in another env.

Michael Hannan,
DB2 Application Performance Specialist
CPT Global Ltd

Edited By:
Michael Hannan[Organization Members] @ Nov 30, 2018 - 03:07 AM (Europe/Berlin)
Michael Hannan[Organization Members] @ Nov 30, 2018 - 03:08 AM (Europe/Berlin)
Michael Hannan[Organization Members] @ Nov 30, 2018 - 03:09 AM (Europe/Berlin)
Michael Hannan[Organization Members] @ Nov 30, 2018 - 03:29 AM (Europe/Berlin)
Michael Hannan[Organization Members] @ Nov 30, 2018 - 03:36 AM (Europe/Berlin)

Jørn Thyssen

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Jørn Thyssen)

Hi Johan,

Feel free to reach out to me! I am based in Denmark, so not far away from Stockholm. My job is essentially to ensure customer have success with IBM Db2 tools :)

I use OCT's "Change Management" feature. During customization the JCL procedure as well as some other required members are generated if you select the following options in IBM TCz:

Change Management database . . . . . . : YES
Enable CM on DB2 Admin Tool primary menu YES
CM Batch JCL procedure name  . . . . . : GOCCM 
Admin Tool PROCLIB . : RSTEST.AOC.ADBC10.PROCLIB  

My sample JCL below runs a compare between some DDL (in SRCIN) against the Db2 catalog and produces a WSL stored in CM.WSL(A000002)

//JOB card
//PROCLIB JCLLIB ORDER=RSTEST.AOC.ADBC10.PROCLIB              
//*                                                           
//GOCCM    EXEC GOCCM,SSID=IDS2,PROFILE=IDS2                  
//GOCCM.REPORT   DD SYSOUT=*                                  
//GOCCM.SYSPRINT DD SYSOUT=*                                  
//GOCCM.ADBMSGS  DD SYSOUT=*                                  
//GOCCM.SYSTDPRT DD SYSOUT=*                                  
//GOCCM.ADBMSGS  DD SYSOUT=*                                  
//GOCCM.ADBDIAG  DD DUMMY                                     
//GOCCM.VALOUT   DD SYSOUT=*                                  
//GOCCM.PARMS DD DISP=SHR,DSN=userid.CM.PARMS(COMPARE)        
//         DD *                                               
 ,PDS_FOR_WSL = 'CM.WSL'                                      
 ,WORKLIST_NAME = 'A000002'                                   
//GOCCM.IGNORES  DD DISP=SHR,DSN=userid.CM.CFG(IGNORES)       
//GOCCM.MASKS    DD DISP=SHR,DSN=userid.CM.CFG(CMPMASK)       
//GOCCM.SRCIN    DD DISP=SHR,DSN=userid.CM.DDL(A000002)       
//GOCCM.ADBTEMPL DD DISP=SHR,DSN=userid.CM.CFG(ADBTEMPL)     

 

DD PARMS are the input parameter documented here:
https://www.ibm.com/support/knowledgecenter/SSAUZ9_12.1.0/topics/adbu_change-man_batch-use-parm-def.html
I store (most) of them in a PDS member as they are normally unchanged between runs:

,ACTION_IMPORT_CHANGE         = 'N'                   
,ACTION_COMPARE               = 'Y'                   
,ACTION_RUN_CHANGE            = 'N'                   
,action_generate_wsl          = 'Y'                   
,CONTENT_OF_APPLY_JOBS        = 'A'                   
,EXISTING_DATA_SET_ACTION     = 'REPLACE'             
,IMPORT_PENDING_CHANGE_ACTION = 'S'                   
,PREFIX_FOR_DATA_SETS         = 'TS5941.CM.TMP'       
,SOURCE_TYPE                  = 'DDL'                 
,REPORT_ONLY_CHANGED_OBJECTS  = 'Y'                   
,REPORT_SUMMARY               = 'Y'                   
,REPORT_OBJECT_TOTALS         = 'Y'                   
,REPORT_TRANSLATION_MASKS     = 'Y'                   
,RUN_SQLID                    = '<NONE>'              
,RUN_REORG_REBUILD            = 'A'                   
,RUN_RUNSTATS                 = 'A'                   
,RUN_REBIND                   = 'Y'                   
,TAKE_AN_IMAGE_COPY           = 'B'                   
,GENERATE_TEMPLATES           = 'Y'                   
,USE_UTILITY_OPTIONS          = 'Y'                   
,UTIL_REORG_SHRLEVEL          = 'C'                   
,UTIL_COPY_SHRLEVEL           = 'C'                   
,UTIL_REORG_INDEX_SHRLEVEL    = 'C'                  

 

DD MASKS is the translation, overwrite and verification masks. If you have used OCT before you are probably already familiar with these. In my case I have a single verification masks that checks if the table space is UTS

VER,TS       :REXX(TSVER,NAME,DBNAME,SEGSIZE,PARTITIONS,MAXPARTITIONS)

but you could add other checks for naming standards, attributes like PRIQTY/SECQTY, AUDIT; DATA CAPTURE CHANGES, etc etc.

DD IGNORES is the fields being ignored, e.g., partitioning or data set sizes. Again, you are probably already familiar with these. I have:

SYSDATABASE:CREATOR                
SYSINDEXES:OWNER                   
SYSPACKAGE:OWNER                   
SYSROUTINES:OWNER                  
SYSSEQUENCES:OWNER                 
SYSTABLES:OWNER                    
SYSTABLESPACE:CREATOR              
SYSTRIGGERS:OWNER                  
SYSVIEWS:OWNER                    

Finally I have my templates in DD ADBTEMPL:

TEMPLATE COPY1 UNIT 3390 DSN  RSTEST.IC.&SSID..&SN..P&PA..&UNIQ. 

 

The job to execute the WSL is structured in a similar way:

//JOB card
//PROCLIB JCLLIB ORDER=RSTEST.AOC.ADBC10.PROCLIB                   
//*                                                                
//GOCCM    EXEC GOCCM,SSID=IDS2,PROFILE=IDS2                       
//GOCCM.REPORT   DD SYSOUT=*                                       
//GOCCM.SYSPRINT DD SYSOUT=*                                       
//GOCCM.ADBMSGS  DD SYSOUT=*                                       
//GOCCM.SYSTDPRT DD SYSOUT=*                                       
//GOCCM.ADBMSGS  DD SYSOUT=*                                       
//GOCCM.VALOUT   DD SYSOUT=*                                       
//GOCCM.ADBDIAG  DD DUMMY                                          
//GOCCM.ADBRPTSM DD SYSOUT=*                                       
//GOCCM.PARMS DD DISP=SHR,DSN=userid.CM.PARMS(RUNWSL)              
//         DD *                                                    
 ,PDS_FOR_WSL = 'CM.WSL'                                           
 ,WORKLIST_NAME = 'A000002'                                       

 

DD PARMS have my parameters for the run:

,ACTION_RUN_WSL               = 'Y'                     
,EXISTING_DATA_SET_ACTION     = 'REPLACE'               
,PREFIX_FOR_DATA_SETS         = 'TS5941.CM.TMP'        

Best regards,

Jørn Thyssen

Rocket Software
77 Fourth Avenue • Waltham, MA • 02451 • USA
E: [login to unmask email] • W: www.rocketsoftware.com 

2018 IBM Champion.

Views are personal. 

Michael Hannan

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Michael Hannan)

Sample Query to detect Column mismatches, this time including missing columns:

  WITH COLS AS                                                        
  (SELECT COLNO CNO                                                    
         ,SMALLINT(COUNT(*) OVER (PARTITION BY TBNAME, NAME)) AS ENVS  
         ,SMALLINT(DENSE_RANK() OVER(PARTITION BY TBNAME, NAME        
            ORDER BY COLTYPE, LENGTH, SCALE, NULLS, DEFAULT, KEYSEQ    
            )) AS RNK1                                                
         ,SMALLINT(DENSE_RANK() OVER(PARTITION BY TBNAME, NAME        
            ORDER BY COLNO)) AS RNK2                                  
         ,SUBSTR(NAME, 1, 18) COL                                      
         ,SUBSTR(TBNAME, 1, 18) TB                                    
         ,SUBSTR(TBCREATOR, 1, 8) CR                                  
         ,COLTYPE TY                                                  
         ,LENGTH  LE                                                  
         ,SCALE   SC                                                  
         ,NULLS   NU                                                  
         ,DEFAULT DF                                                  
         ,KEYSEQ  PK                                                  
         ,CREATEDTS CTS                                                
         ,ALTEREDTS ATS                                                
         ,SMALLINT(DENSE_RANK() OVER(ORDER BY TBCREATOR)) ENVRNK      
     FROM SYSIBM.SYSCOLUMNS C                                          
    WHERE TBCREATOR LIKE '__PUDBA'                                    
    AND  (TBCREATOR LIKE 'A%'                                          
      OR  TBCREATOR LIKE 'B%'                                          
      OR  TBCREATOR LIKE 'K%'                                          
         )                                                            
    AND   EXISTS (SELECT 1 FROM SYSIBM.SYSTABLES T                    
                   WHERE T.TYPE IN ('T','M')                          
                   AND   T.CREATOR = C.TBCREATOR                      
                   AND   T.NAME    = C.TBNAME )                        
  )                                                                    
  ,COLS2 AS                                                            
  (SELECT C.*                                                          
         ,MAX(ENVRNK) OVER() TENV                                      
         ,SMALLINT(MAX(RNK1) OVER(PARTITION BY TB, COL)) AS D1        
         ,SMALLINT(MAX(RNK2) OVER(PARTITION BY TB, COL)) AS D2        
     FROM COLS C                                                      
  )                                                                    
  SELECT * FROM COLS2                                                  
   WHERE (D1 > 1 OR D2 > 1 OR ENVS < TENV)                            
   ORDER BY TB, CNO, CR                                                
   FETCH FIRST 1000 ROWS ONLY                                          
   WITH UR;                                                            

Michael Hannan,
DB2 Application Performance Specialist
CPT Global Ltd

Johan Sundborg

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Michael Hannan)

Hi and thanks for a quick and rather extensive reply!

I agree that one should be able to put together a comparison tool of your own, it's impressive that you have whooped up quite a lot of features. From my perspective it feels like quite a challenge to create a panel routine that do a full compare of any Db2-object between environments and Db2-subsystems, preferably grabbing all catalog columns that are relevant (maybe not Creator or PRI-/SECQTY). Although, you could probably select most (or all) DDL from the catalog tables, save to file for each environment and then do a simple comparison, source-file with the rest. That approach could give you a collective report and also a delta for each target environment.

So far so good, then you "just" have to apply the delta and you're home safe. This should preferable be done as a part of the panel choice (or a step 2).

As we all know, alters can result in numerous restrictive statuses on the Db2-object. A tricky part could be to incorporate the solution for each status into the panel routine to minimize any downtime. One way to do this could be to be proactive and look into every possible situation where a status might occur and have the routine solve it in a tested and optimal way. That would however be very time consuming to setup and likely sensitive to version changes in Db2.

Another solution could be to be reactive, meaning you run the compare, then the apply and then a display of the objects you fiddled with. The routine then checks the display output and solve any unwanted status by generating one or more utility jobs from templates and runs them (this would also require some work, but likely less then the previous solution).

As you mention, an issue with home made solutions is dependencies on key employees. You don't want to get stranded if the one who build all the fancy SQL and jobs for example retire (could possible .

I'm not completely sure what you mean with "full change management tool" but OCT from IBM can make comparisons and create a delta job with required utilities. After reviewing the delta job, you just run it and your done. The drawbacks is that you can only compare one target environment and one source environment at the time; the delta job is good but not 100% great and the tool costs "some" money annually. That's why we both consider it and also look for other competitive solutions :)

Regards
Johan Sundborg, Db2 DBA


In Reply to Michael Hannan:

Sample Query to detect Column mismatches, this time including missing columns:

  WITH COLS AS                                                        
  (SELECT COLNO CNO                                                    
         ,SMALLINT(COUNT(*) OVER (PARTITION BY TBNAME, NAME)) AS ENVS  
         ,SMALLINT(DENSE_RANK() OVER(PARTITION BY TBNAME, NAME        
            ORDER BY COLTYPE, LENGTH, SCALE, NULLS, DEFAULT, KEYSEQ    
            )) AS RNK1                                                
         ,SMALLINT(DENSE_RANK() OVER(PARTITION BY TBNAME, NAME        
            ORDER BY COLNO)) AS RNK2                                  
         ,SUBSTR(NAME, 1, 18) COL                                      
         ,SUBSTR(TBNAME, 1, 18) TB                                    
         ,SUBSTR(TBCREATOR, 1, 8) CR                                  
         ,COLTYPE TY                                                  
         ,LENGTH  LE                                                  
         ,SCALE   SC                                                  
         ,NULLS   NU                                                  
         ,DEFAULT DF                                                  
         ,KEYSEQ  PK                                                  
         ,CREATEDTS CTS                                                
         ,ALTEREDTS ATS                                                
         ,SMALLINT(DENSE_RANK() OVER(ORDER BY TBCREATOR)) ENVRNK      
     FROM SYSIBM.SYSCOLUMNS C                                          
    WHERE TBCREATOR LIKE '__PUDBA'                                    
    AND  (TBCREATOR LIKE 'A%'                                          
      OR  TBCREATOR LIKE 'B%'                                          
      OR  TBCREATOR LIKE 'K%'                                          
         )                                                            
    AND   EXISTS (SELECT 1 FROM SYSIBM.SYSTABLES T                    
                   WHERE T.TYPE IN ('T','M')                          
                   AND   T.CREATOR = C.TBCREATOR                      
                   AND   T.NAME    = C.TBNAME )                        
  )                                                                    
  ,COLS2 AS                                                            
  (SELECT C.*                                                          
         ,MAX(ENVRNK) OVER() TENV                                      
         ,SMALLINT(MAX(RNK1) OVER(PARTITION BY TB, COL)) AS D1        
         ,SMALLINT(MAX(RNK2) OVER(PARTITION BY TB, COL)) AS D2        
     FROM COLS C                                                      
  )                                                                    
  SELECT * FROM COLS2                                                  
   WHERE (D1 > 1 OR D2 > 1 OR ENVS < TENV)                            
   ORDER BY TB, CNO, CR                                                
   FETCH FIRST 1000 ROWS ONLY                                          
   WITH UR;                                                            

Michael Hannan,
DB2 Application Performance Specialist
CPT Global Ltd

Bharath Nunepalli

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Johan Sundborg)

Johan,

I have developed my own OCT using REXX.

The input is the source and target databases (both can be present in same or different Db2 sub-systems) and the output is loaded into a Db2 table for future reference. The Db2 table holding the comparison details has the below columns.
SRC_DB
TGT_DB
TS_NAME
TS_MISS_IN_TGT
TS_TYPE_MISMATCH
TB_NAME
TB_MISS_IN_TGT
IX_NAME
IX_KEY_NAME
IX_MISS_IN_TGT
IX_KEY_MISMATCH
CL_NAME
CL_MISS_IN_TGT
CL_TYPE_MISMATCH
CL_LEN_MISMATCH
CL_SCL_MISMATCH
CL_NUL_MISMATCH
CL_SEQ_MISMATCH
ROW_ENTRY_TS

 

Bharath Nunepalli,
Senior DB2 DBA.

Michael Hannan

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Johan Sundborg)



In Reply to Johan Sundborg:

I'm not completely sure what you mean with "full change management tool" but OCT from IBM can make comparisons and create a delta job with required utilities. After reviewing the delta job, you just run it and your done. The drawbacks is that you can only compare one target environment and one source environment at the time; the delta job is good but not 100% great and the tool costs "some" money annually. That's why we both consider it and also look for other competitive solutions :)

By full Change Management, I mean to generate all the impacted DDLs, Reorg Jobs, Backups, etc. for a change. While I have built Object compare reports, and also built DDL Generators, I did not try to build the Reorg generators as well. I also did not try to build ALTERs etc. for adding columns, just DROPs and CREATEs type DDL, mainly intended for cloning an environment. Was years ago so never handled XML or Temporal tables, but handled dependent Views, Aliases, Synonyms (back then), Grants, etc. Generating dependent objects is not that tough. The problem with building your own tools, is advancements in DB2 each release, mean more new features to be handled, so that means time for mods and testing, as errors creep in. Hard to test all DB2 features if your shop does not use them.

I also built DB2 Stats compares, as migrating Prod Stats to another env is critical to getting and predicting the same access paths as Production. Also built something to generate SQL to migrate Prod Stats to other envs. Just a set of Stats UPDATEs and DELETE/INSERTs for SYSCOLDIST Stats. There are other possibilities for greater efficiency I guess, such as LOAD all Stats to special dedicated Stats tables using Cross Loader, then run the much smaller number of SQLs to apply them to the real Catalog tables, or use Static SQLs in a program. 

The advantage of a compare report, using SQL OLAP, is can compare any number of environments in the one run, not needing to designate a source and a target. The key feature being the ability to sequentially assign a rank number to each distinct value of a multi-column comparison key. I will probably extend my Table columns compare report, to be able to do: Indexes and Keys (done already), Tablespaces, Tables, Primary Keys, Foreign Keys, etc. Views and Aliases on basis on present or missing, and maybe can come up with something for View text as well (with multiple blanks removal) and possible needing to replace certain Authids with markers. Just depends on what things have a requirement to compare, and which selected column values need to match between environments.

If a tool can compare only pairs, then need to designate one env as the baseline, and then compare each other env to it, so that is not so tough really. A smart compare should allow env A to be compared to each of B, C, D, E and F.

Automated Reorg tools don't always seem to have the flexibility to generate Reorgs based on differing requirements profiles for different tables, so I am tempted to build my own queries, firstly to determine when tables need Reorg, when to collect Runstats (without being tied to Reorg), and certainly not tying REBINDs to those things.

Its probably crazy to build your own tools, but does not cost real money (only time), and at least I can try to make my own tools do what I want them to, rather than what the vendor decided. So it is a case of weighing up how much a tool costs and how much time would it take to build your own, and do you time available. For just comparing things to check for inconsistencies, I always had enough time to build my own and years ago needed it at a somewhat chaotic site, because the task is not so difficult. Migrating DB2 Stats was also not so difficult. I build automation for things I need with high frequency, e.g. populating DSN_VIRTUAL_INDEXES to try a proposed index. Manually doing it is a bit too much effort. 

Michael Hannan,
DB2 Application Performance Specialist
CPT Global Ltd

Edited By:
Michael Hannan[Organization Members] @ Dec 05, 2018 - 04:06 AM (Europe/Berlin)

Johan Sundborg

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Bharath Nunepalli)

Hi Bharath,

REXX is great and absolutely a valid option when comparing Db2 objects and generating reports or input for a table, such as the one you describe.

We have something similar setup in a home made ISPF panel where you enter target/source (database) prefix and the environments to be compared. The job behind the panel then selects from catalog tables, unloads to file, compare files and creates a result file with the mis-matches. This helps us a lot in keeping certain environments in sync.

However, I don't think REXX have much of an advantage over regular batch when it comes to making these kinds of comparisons, creating delta, etc. it's probably more of a personal preference.

Regards
Johan Sundborg, Db2 DBA

In Reply to Bharath Nunepalli:

Johan,

I have developed my own OCT using REXX.

The input is the source and target databases (both can be present in same or different Db2 sub-systems) and the output is loaded into a Db2 table for future reference. The Db2 table holding the comparison details has the below columns.
SRC_DB
TGT_DB
TS_NAME
TS_MISS_IN_TGT
TS_TYPE_MISMATCH
TB_NAME
TB_MISS_IN_TGT
IX_NAME
IX_KEY_NAME
IX_MISS_IN_TGT
IX_KEY_MISMATCH
CL_NAME
CL_MISS_IN_TGT
CL_TYPE_MISMATCH
CL_LEN_MISMATCH
CL_SCL_MISMATCH
CL_NUL_MISMATCH
CL_SEQ_MISMATCH
ROW_ENTRY_TS

 

Bharath Nunepalli,
Senior DB2 DBA.

Johan Sundborg

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Jørn Thyssen)

Hi Jørn,

That's very interesting. I was hoping, but wasn't completely sure, that IBM OCT had some kind of batch feature or support. The top 4 parameters you show

In Reply to Jørn Thyssen:

Change Management database . . . . . . : YES
Enable CM on DB2 Admin Tool primary menu YES 
CM Batch JCL procedure name  . . . . . : GOCCM  
Admin Tool PROCLIB . : RSTEST.AOC.ADBC10.PROCLIB   

feels like they are a part of the setup of either OCT or Admin Tool (or both?), which would be something for our Db2-technical team to look into.

To give some more background: We had OCT in the company for many years due to it being included in a tool-deal with IBM where we also got Admin Tool and more. We (I say "we" even though I wasn't at the company at the time) however didn't use OCT and last spring decided to uninstall it and revoke our license. Now the company have decided to go full agile and as a part of that, we are interested in either reinstalling OCT or something similar to speed up our time to market. So we currently don't have OCT installed, but we do have a trial licence on standby that we are going to run when all participants feel ready.

Your JCL looks quite straight forward and yes, I'm familiar with the parm/mask/ignore. Can you with the batch feature for example compare Db2-catalog-env1 with Db2-catalog-env2 for a prespecified database/tablespace? similar to the manual use of OCT when you specify what Db2-objects that should be included in the source/target-compare.

It would be interesting to chat some more about what can be done with automation and OCT.

By the way, one of my colleagues mentioned "Zowe" as a potential candidate that possible could include IBM OCT and more as plugins. Have you heard anything about that collaboration and what one can expect regarding the change management parts?

https://www.openmainframeproject.org/projects/zowe

https://www.youtube.com/watch?v=0s9NhrnYzUc

Regards

Johan Sundborg, Db2 DBA

In Reply to Jørn Thyssen:

Hi Johan,

Feel free to reach out to me! I am based in Denmark, so not far away from Stockholm. My job is essentially to ensure customer have success with IBM Db2 tools :)

I use OCT's "Change Management" feature. During customization the JCL procedure as well as some other required members are generated if you select the following options in IBM TCz:

Change Management database . . . . . . : YES
Enable CM on DB2 Admin Tool primary menu YES
CM Batch JCL procedure name  . . . . . : GOCCM 
Admin Tool PROCLIB . : RSTEST.AOC.ADBC10.PROCLIB  

My sample JCL below runs a compare between some DDL (in SRCIN) against the Db2 catalog and produces a WSL stored in CM.WSL(A000002)

//JOB card
//PROCLIB JCLLIB ORDER=RSTEST.AOC.ADBC10.PROCLIB              
//*                                                           
//GOCCM    EXEC GOCCM,SSID=IDS2,PROFILE=IDS2                  
//GOCCM.REPORT   DD SYSOUT=*                                  
//GOCCM.SYSPRINT DD SYSOUT=*                                  
//GOCCM.ADBMSGS  DD SYSOUT=*                                  
//GOCCM.SYSTDPRT DD SYSOUT=*                                  
//GOCCM.ADBMSGS  DD SYSOUT=*                                  
//GOCCM.ADBDIAG  DD DUMMY                                     
//GOCCM.VALOUT   DD SYSOUT=*                                  
//GOCCM.PARMS DD DISP=SHR,DSN=userid.CM.PARMS(COMPARE)        
//         DD *                                               
 ,PDS_FOR_WSL = 'CM.WSL'                                      
 ,WORKLIST_NAME = 'A000002'                                   
//GOCCM.IGNORES  DD DISP=SHR,DSN=userid.CM.CFG(IGNORES)       
//GOCCM.MASKS    DD DISP=SHR,DSN=userid.CM.CFG(CMPMASK)       
//GOCCM.SRCIN    DD DISP=SHR,DSN=userid.CM.DDL(A000002)       
//GOCCM.ADBTEMPL DD DISP=SHR,DSN=userid.CM.CFG(ADBTEMPL)     

 

DD PARMS are the input parameter documented here:
https://www.ibm.com/support/knowledgecenter/SSAUZ9_12.1.0/topics/adbu_change-man_batch-use-parm-def.html
I store (most) of them in a PDS member as they are normally unchanged between runs:

,ACTION_IMPORT_CHANGE         = 'N'                   
,ACTION_COMPARE               = 'Y'                   
,ACTION_RUN_CHANGE            = 'N'                   
,action_generate_wsl          = 'Y'                   
,CONTENT_OF_APPLY_JOBS        = 'A'                   
,EXISTING_DATA_SET_ACTION     = 'REPLACE'             
,IMPORT_PENDING_CHANGE_ACTION = 'S'                   
,PREFIX_FOR_DATA_SETS         = 'TS5941.CM.TMP'       
,SOURCE_TYPE                  = 'DDL'                 
,REPORT_ONLY_CHANGED_OBJECTS  = 'Y'                   
,REPORT_SUMMARY               = 'Y'                   
,REPORT_OBJECT_TOTALS         = 'Y'                   
,REPORT_TRANSLATION_MASKS     = 'Y'                   
,RUN_SQLID                    = ''              
,RUN_REORG_REBUILD            = 'A'                   
,RUN_RUNSTATS                 = 'A'                   
,RUN_REBIND                   = 'Y'                   
,TAKE_AN_IMAGE_COPY           = 'B'                   
,GENERATE_TEMPLATES           = 'Y'                   
,USE_UTILITY_OPTIONS          = 'Y'                   
,UTIL_REORG_SHRLEVEL          = 'C'                   
,UTIL_COPY_SHRLEVEL           = 'C'                   
,UTIL_REORG_INDEX_SHRLEVEL    = 'C'                  

 

DD MASKS is the translation, overwrite and verification masks. If you have used OCT before you are probably already familiar with these. In my case I have a single verification masks that checks if the table space is UTS

VER,TS       :REXX(TSVER,NAME,DBNAME,SEGSIZE,PARTITIONS,MAXPARTITIONS)

but you could add other checks for naming standards, attributes like PRIQTY/SECQTY, AUDIT; DATA CAPTURE CHANGES, etc etc.

DD IGNORES is the fields being ignored, e.g., partitioning or data set sizes. Again, you are probably already familiar with these. I have:

SYSDATABASE:CREATOR                
SYSINDEXES:OWNER                   
SYSPACKAGE:OWNER                   
SYSROUTINES:OWNER                  
SYSSEQUENCES:OWNER                 
SYSTABLES:OWNER                    
SYSTABLESPACE:CREATOR              
SYSTRIGGERS:OWNER                  
SYSVIEWS:OWNER                    

Finally I have my templates in DD ADBTEMPL:

TEMPLATE COPY1 UNIT 3390 DSN  RSTEST.IC.&SSID..&SN..P&PA..&UNIQ. 

 

The job to execute the WSL is structured in a similar way:

//JOB card
//PROCLIB JCLLIB ORDER=RSTEST.AOC.ADBC10.PROCLIB                   
//*                                                                
//GOCCM    EXEC GOCCM,SSID=IDS2,PROFILE=IDS2                       
//GOCCM.REPORT   DD SYSOUT=*                                       
//GOCCM.SYSPRINT DD SYSOUT=*                                       
//GOCCM.ADBMSGS  DD SYSOUT=*                                       
//GOCCM.SYSTDPRT DD SYSOUT=*                                       
//GOCCM.ADBMSGS  DD SYSOUT=*                                       
//GOCCM.VALOUT   DD SYSOUT=*                                       
//GOCCM.ADBDIAG  DD DUMMY                                          
//GOCCM.ADBRPTSM DD SYSOUT=*                                       
//GOCCM.PARMS DD DISP=SHR,DSN=userid.CM.PARMS(RUNWSL)              
//         DD *                                                    
 ,PDS_FOR_WSL = 'CM.WSL'                                           
 ,WORKLIST_NAME = 'A000002'                                       

 

DD PARMS have my parameters for the run:

,ACTION_RUN_WSL               = 'Y'                     
,EXISTING_DATA_SET_ACTION     = 'REPLACE'               
,PREFIX_FOR_DATA_SETS         = 'TS5941.CM.TMP'        

Best regards,

Jørn Thyssen

Rocket Software
77 Fourth Avenue • Waltham, MA • 02451 • USA
E: [login to unmask email] • W: www.rocketsoftware.com 

2018 IBM Champion.

Views are personal. 

Chris Hoelscher

Db2 z/OS v12 - Object Comparison Tool automation
(in response to Johan Sundborg)
The one dev team that does frequent object compares uses RC/Compare

Chris Hoelscher
Technology Architect, Database Infrastructure Services
Technology Solution Services
Humana Inc.
123 East Main Street
Louisville, KY 40202
Humana.com
(502) 476-2538 or 407-7266

From: Johan Sundborg <[login to unmask email]>
Sent: Wednesday, December 5, 2018 3:16 AM
To: [login to unmask email]
Subject: [DB2-L] - RE: Db2 z/OS v12 - Object Comparison Tool automation


Hi Bharath,

REXX is great and absolutely a valid option when comparing Db2 objects and generating reports or input for a table, such as the one you describe.

We have something similar setup in a home made ISPF panel where you enter target/source (database) prefix and the environments to be compared. The job behind the panel then selects from catalog tables, unloads to file, compare files and creates a result file with the mis-matches. This helps us a lot in keeping certain environments in sync.

However, I don't think REXX have much of an advantage over regular batch when it comes to making these kinds of comparisons, creating delta, etc. it's probably more of a personal preference.

Regards
Johan Sundborg, Db2 DBA

In Reply to Bharath Nunepalli:

Johan,

I have developed my own OCT using REXX.

The input is the source and target databases (both can be present in same or different Db2 sub-systems) and the output is loaded into a Db2 table for future reference. The Db2 table holding the comparison details has the below columns.
SRC_DB
TGT_DB
TS_NAME
TS_MISS_IN_TGT
TS_TYPE_MISMATCH
TB_NAME
TB_MISS_IN_TGT
IX_NAME
IX_KEY_NAME
IX_MISS_IN_TGT
IX_KEY_MISMATCH
CL_NAME
CL_MISS_IN_TGT
CL_TYPE_MISMATCH
CL_LEN_MISMATCH
CL_SCL_MISMATCH
CL_NUL_MISMATCH
CL_SEQ_MISMATCH
ROW_ENTRY_TS



Bharath Nunepalli,
Senior DB2 DBA.

-----End Original Message-----

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

Humana Inc. and its subsidiaries comply with applicable Federal civil rights laws and
do not discriminate on the basis of race, color, national origin, age, disability, sex,
sexual orientation, gender identity, or religion. Humana Inc. and its subsidiaries do not
exclude people or treat them differently because of race, color, national origin, age,
disability, sex, sexual orientation, gender identity, or religion.

English: ATTENTION: If you do not speak English, language assistance services, free
of charge, are available to you. Call 1‐877‐320‐1235 (TTY: 711).

Español (Spanish): ATENCIÓN: Si habla español, tiene a su disposición servicios
gratuitos de asistencia lingüística. Llame al 1‐877‐320‐1235 (TTY: 711).

繁體中文(Chinese):注意:如果您使用繁體中文,您可以免費獲得語言援助
服務。請致電 1‐877‐320‐1235 (TTY: 711)。

Kreyòl Ayisyen (Haitian Creole): ATANSION: Si w pale Kreyòl Ayisyen, gen sèvis èd
pou lang ki disponib gratis pou ou. Rele 1‐877‐320‐1235 (TTY: 711).

Polski (Polish): UWAGA: Jeżeli mówisz po polsku, możesz skorzystać z bezpłatnej
pomocy językowej. Zadzwoń pod numer 1‐877‐320‐1235 (TTY: 711).

한국어 (Korean): 주의: 한국어를 사용하시는 경우, 언어 지원 서비스를 무료로
이용하실 수 있습니다. 1‐877‐320‐1235 (TTY: 711)번으로 전화해 주십시오.

J&#248;rn Thyssen

RE: Db2 z/OS v12 - Object Comparison Tool automation [ADish]
(in response to Johan Sundborg)

Hi Johan,

The 4 parameters I showed is part of the customization. So your Db2 tools sysprog will have to check if they were set appropriately and tell you the name of the PROCLIB, so you can point to it in your JCL.

 

Yes, you can compare across environments assuming you have DDF connection.
Your CM Batch parameter would look like this:
Instead of SOURCE_TYPE                  = 'DDL'      you would use:

,SOURCE_TYPE = 'USER'
,SOURCE_LOCATION = 'DDF location name for source'
,TARGET_TYPE = 'AUTO'
,TARGET_LOCATION = 'DDF location for target'

And instead of SRCIN pointing to a dataset with DDL you would have:
//SRCIN DD *
 TYPE='DB',NAME='DSN81110';
 TYPE = 'TB',QUAL='DSN81110',NAME='DEPT';
You can of course have multiple lines and type supports all kinds of object types. The syntax is the same as used by GEN.

You can write your own wrapping on top of Admin&Compare's CMBATCH function: I have customers:

  • writing their own ISPF panels that build CM Batch jobs
  • writing a web application that submits CM Batch jobs
  • submit CM Batch jobs through REST API using z/OSMF REST capabilities


AD:
If you don't want to write your own tooling IBM have a new solution IBM Db2 DevOps Experience that allows developers to self-service provision applications, modify DDL, apply the changes and finally submit the change for DBA approval. Initially only target for the developer cycle, but we often get the feedback that DBAs want to use the same application to deploy the DDL into higher environments, e.g, test and production.
IBM Db2 DevOps Experience is built on top of Zowe, but it does cost $$$.

I will present this solution at the Nordic Db2 Day in March 2019 in Stockholm, but you can also reach out to your local IBM team to get more information.

Best regards,

Jørn Thyssen

Rocket Software
77 Fourth Avenue • Waltham, MA • 02451 • USA
E: [login to unmask email] • W: www.rocketsoftware.com 

2018 IBM Champion.

Views are personal. 

Johan Sundborg

RE: Db2 z/OS v12 - Object Comparison Tool automation
(in response to Chris Hoelscher)

Hi Chris,

I haven't heard of RC/Compare before but from what I can gather from youtube and CAs homepage RC/Compare is quite similar to IMB OCT, with a few differences. How would you judge the pricing? high/medium/low?

Since CA is a part of the Zowe initiative, will RC/Compare be available as a plugin there in the future?

Regards
Johan Sundborg, Db2 DBA

In Reply to Chris Hoelscher:

The one dev team that does frequent object compares uses RC/Compare

Chris Hoelscher
Technology Architect, Database Infrastructure Services
Technology Solution Services
Humana Inc.
123 East Main Street
Louisville, KY 40202
Humana.com
(502) 476-2538 or 407-7266

From: Johan Sundborg <[login to unmask email]>
Sent: Wednesday, December 5, 2018 3:16 AM
To: [login to unmask email]
Subject: [DB2-L] - RE: Db2 z/OS v12 - Object Comparison Tool automation


Hi Bharath,

REXX is great and absolutely a valid option when comparing Db2 objects and generating reports or input for a table, such as the one you describe.

We have something similar setup in a home made ISPF panel where you enter target/source (database) prefix and the environments to be compared. The job behind the panel then selects from catalog tables, unloads to file, compare files and creates a result file with the mis-matches. This helps us a lot in keeping certain environments in sync.

However, I don't think REXX have much of an advantage over regular batch when it comes to making these kinds of comparisons, creating delta, etc. it's probably more of a personal preference.

Regards
Johan Sundborg, Db2 DBA

In Reply to Bharath Nunepalli:

Johan,

I have developed my own OCT using REXX.

The input is the source and target databases (both can be present in same or different Db2 sub-systems) and the output is loaded into a Db2 table for future reference. The Db2 table holding the comparison details has the below columns.
SRC_DB
TGT_DB
TS_NAME
TS_MISS_IN_TGT
TS_TYPE_MISMATCH
TB_NAME
TB_MISS_IN_TGT
IX_NAME
IX_KEY_NAME
IX_MISS_IN_TGT
IX_KEY_MISMATCH
CL_NAME
CL_MISS_IN_TGT
CL_TYPE_MISMATCH
CL_LEN_MISMATCH
CL_SCL_MISMATCH
CL_NUL_MISMATCH
CL_SEQ_MISMATCH
ROW_ENTRY_TS



Bharath Nunepalli,
Senior DB2 DBA.

-----End Original Message-----

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

Humana Inc. and its subsidiaries comply with applicable Federal civil rights laws and
do not discriminate on the basis of race, color, national origin, age, disability, sex,
sexual orientation, gender identity, or religion. Humana Inc. and its subsidiaries do not
exclude people or treat them differently because of race, color, national origin, age,
disability, sex, sexual orientation, gender identity, or religion.

English: ATTENTION: If you do not speak English, language assistance services, free
of charge, are available to you. Call 1‐877‐320‐1235 (TTY: 711).

Español (Spanish): ATENCIÓN: Si habla español, tiene a su disposición servicios
gratuitos de asistencia lingüística. Llame al 1‐877‐320‐1235 (TTY: 711).

繁體中文(Chinese):注意:如果您使用繁體中文,您可以免費獲得語言援助
服務。請致電 1‐877‐320‐1235 (TTY: 711)。

Kreyòl Ayisyen (Haitian Creole): ATANSION: Si w pale Kreyòl Ayisyen, gen sèvis èd
pou lang ki disponib gratis pou ou. Rele 1‐877‐320‐1235 (TTY: 711).

Polski (Polish): UWAGA: Jeżeli mówisz po polsku, możesz skorzystać z bezpłatnej
pomocy językowej. Zadzwoń pod numer 1‐877‐320‐1235 (TTY: 711).

한국어 (Korean): 주의: 한국어를 사용하시는 경우, 언어 지원 서비스를 무료로
이용하실 수 있습니다. 1‐877‐320‐1235 (TTY: 711)번으로 전화해 주십시오.

Chris Hoelscher

Db2 z/OS v12 - Object Comparison Tool automation
(in response to Johan Sundborg)
I am not privy to pricing – rc/compare is a subset of rc/migrate – you must have rc/migrate to run rc/compare

I also know nothing about the zowe initiative

Chris Hoelscher
Technology Architect, Database Infrastructure Services
Technology Solution Services
Humana Inc.
123 East Main Street
Louisville, KY 40202
Humana.com
(502) 476-2538 or 407-7266

From: Johan Sundborg <[login to unmask email]>
Sent: Thursday, December 6, 2018 9:08 AM
To: [login to unmask email]
Subject: [DB2-L] - RE: Db2 z/OS v12 - Object Comparison Tool automation


Hi Chris,

I haven't heard of RC/Compare before but from what I can gather from youtube and CAs homepage RC/Compare is quite similar to IMB OCT, with a few differences. How would you judge the pricing? high/medium/low?

Since CA is a part of the Zowe initiative, will RC/Compare be available as a plugin there in the future?

Regards
Johan Sundborg, Db2 DBA

In Reply to Chris Hoelscher:
The one dev team that does frequent object compares uses RC/Compare

Chris Hoelscher
Technology Architect, Database Infrastructure Services
Technology Solution Services
Humana Inc.
123 East Main Street
Louisville, KY 40202
Humana.com
(502) 476-2538 or 407-7266

From: Johan Sundborg
Sent: Wednesday, December 5, 2018 3:16 AM
To: [login to unmask email]<mailto:[login to unmask email]>
Subject: [DB2-L] - RE: Db2 z/OS v12 - Object Comparison Tool automation


Hi Bharath,

REXX is great and absolutely a valid option when comparing Db2 objects and generating reports or input for a table, such as the one you describe.

We have something similar setup in a home made ISPF panel where you enter target/source (database) prefix and the environments to be compared. The job behind the panel then selects from catalog tables, unloads to file, compare files and creates a result file with the mis-matches. This helps us a lot in keeping certain environments in sync.

However, I don't think REXX have much of an advantage over regular batch when it comes to making these kinds of comparisons, creating delta, etc. it's probably more of a personal preference.

Regards
Johan Sundborg, Db2 DBA

In Reply to Bharath Nunepalli:

Johan,

I have developed my own OCT using REXX.

The input is the source and target databases (both can be present in same or different Db2 sub-systems) and the output is loaded into a Db2 table for future reference. The Db2 table holding the comparison details has the below columns.
SRC_DB
TGT_DB
TS_NAME
TS_MISS_IN_TGT
TS_TYPE_MISMATCH
TB_NAME
TB_MISS_IN_TGT
IX_NAME
IX_KEY_NAME
IX_MISS_IN_TGT
IX_KEY_MISMATCH
CL_NAME
CL_MISS_IN_TGT
CL_TYPE_MISMATCH
CL_LEN_MISMATCH
CL_SCL_MISMATCH
CL_NUL_MISMATCH
CL_SEQ_MISMATCH
ROW_ENTRY_TS



Bharath Nunepalli,
Senior DB2 DBA.

-----End Original Message-----

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

Humana Inc. and its subsidiaries comply with applicable Federal civil rights laws and
do not discriminate on the basis of race, color, national origin, age, disability, sex,
sexual orientation, gender identity, or religion. Humana Inc. and its subsidiaries do not
exclude people or treat them differently because of race, color, national origin, age,
disability, sex, sexual orientation, gender identity, or religion.

English: ATTENTION: If you do not speak English, language assistance services, free
of charge, are available to you. Call 1‐877‐320‐1235 (TTY: 711).

Español (Spanish): ATENCIÓN: Si habla español, tiene a su disposición servicios
gratuitos de asistencia lingüística. Llame al 1‐877‐320‐1235 (TTY: 711).

繁體中文(Chinese):注意:如果您使用繁體中文,您可以免費獲得語言援助
服務。請致電 1‐877‐320‐1235 (TTY: 711)。

Kreyòl Ayisyen (Haitian Creole): ATANSION: Si w pale Kreyòl Ayisyen, gen sèvis èd
pou lang ki disponib gratis pou ou. Rele 1‐877‐320‐1235 (TTY: 711).

Polski (Polish): UWAGA: Jeżeli mówisz po polsku, możesz skorzystać z bezpłatnej
pomocy językowej. Zadzwoń pod numer 1‐877‐320‐1235 (TTY: 711).

한국어 (Korean): 주의: 한국어를 사용하시는 경우, 언어 지원 서비스를 무료로
이용하실 수 있습니다. 1‐877‐320‐1235 (TTY: 711)번으로 전화해 주십시오.

-----End Original Message-----

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

Humana Inc. and its subsidiaries comply with applicable Federal civil rights laws and
do not discriminate on the basis of race, color, national origin, age, disability, sex,
sexual orientation, gender identity, or religion. Humana Inc. and its subsidiaries do not
exclude people or treat them differently because of race, color, national origin, age,
disability, sex, sexual orientation, gender identity, or religion.

English: ATTENTION: If you do not speak English, language assistance services, free
of charge, are available to you. Call 1‐877‐320‐1235 (TTY: 711).

Español (Spanish): ATENCIÓN: Si habla español, tiene a su disposición servicios
gratuitos de asistencia lingüística. Llame al 1‐877‐320‐1235 (TTY: 711).

繁體中文(Chinese):注意:如果您使用繁體中文,您可以免費獲得語言援助
服務。請致電 1‐877‐320‐1235 (TTY: 711)。

Kreyòl Ayisyen (Haitian Creole): ATANSION: Si w pale Kreyòl Ayisyen, gen sèvis èd
pou lang ki disponib gratis pou ou. Rele 1‐877‐320‐1235 (TTY: 711).

Polski (Polish): UWAGA: Jeżeli mówisz po polsku, możesz skorzystać z bezpłatnej
pomocy językowej. Zadzwoń pod numer 1‐877‐320‐1235 (TTY: 711).

한국어 (Korean): 주의: 한국어를 사용하시는 경우, 언어 지원 서비스를 무료로
이용하실 수 있습니다. 1‐877‐320‐1235 (TTY: 711)번으로 전화해 주십시오.