FDR Backup - Recovery

Steve Grimes

FDR Backup - Recovery
Hello, DB2 V7.1, Z/os 1.4 here.

We need to upgrade our Peopletools software (and tables) in our production
DB2 subsystem, and lack backup/recovery tools that could help us. The one
"good" thing we may have is availble down time. So, we're thinking of the
following:

1) Shut down everything.
2) FDR everything to tape. (Our typical weekend backup.) (This is our
fail-safe fall-back.)
3) FDR the DB2 production subsystem volumes to tape.
(Volumes were identified by Stogroup definitions and dataset HLQ's of
Subsys + "CAT", "APP" or "PT".
We think this picks up the Catalog, App tables & Log files. I'm not
sure about the directory(?) and
what else we may be missing.
I'm thinking we should also include the MVS User Catalog for the HLQ's
involved, because the
upgrade will drop and redefine tables, and they could move.
I'm most worried about what I could be missing here.
4) Bring up just enough too...
5) Upgrade PeopleTools
6) Test test test

If failure occurs, shut down everything and fall back to volumes backed up
in step 3.

Any thoughts or comments? For instance, should I be concerned about
emptying the DB2 active logs, so that they won't fill up during testing and
mount tapes, which in turn update entries in our tape management system?
(I need to read more about DB2 Log Suspend also, to see if we need to do
this. I'm hoping that shutting down DB2 in step 1 takes care of this.)

Many thanks in advance!

Stg

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

cliff boley

Re: FDR Backup - Recovery
(in response to Steve Grimes)
Steve,
You must have imagecopy and recover utilities.
What it looks like your trying to do is the same a setting up for a disaster
recovery.
What about imagcopy you peopletools tablespaces using listdef then
imagecopy the catalog and directory. Next archive the logs/bsds. Export the
DB2 ICF catalog, then FDR all of your DB2 distribution libraries.
If your upgrade fails and you need to fall back you could use your disaster
recovery procedures
to recover.

How does this sound?
cliff:-)


-----Original Message-----
From: [login to unmask email] [mailto:[login to unmask email]

Sent: Monday, December 13, 2004 9:55 AM
To: [login to unmask email]
Subject: FDR Backup - Recovery


Hello, DB2 V7.1, Z/os 1.4 here.

We need to upgrade our Peopletools software (and tables) in our production
DB2 subsystem, and lack backup/recovery tools that could help us. The one
"good" thing we may have is availble down time. So, we're thinking of the
following:

1) Shut down everything.
2) FDR everything to tape. (Our typical weekend backup.) (This is our
fail-safe fall-back.)
3) FDR the DB2 production subsystem volumes to tape.
(Volumes were identified by Stogroup definitions and dataset HLQ's of
Subsys + "CAT", "APP" or "PT".
We think this picks up the Catalog, App tables & Log files. I'm not
sure about the directory(?) and
what else we may be missing.
I'm thinking we should also include the MVS User Catalog for the HLQ's
involved, because the
upgrade will drop and redefine tables, and they could move.
I'm most worried about what I could be missing here.
4) Bring up just enough too...
5) Upgrade PeopleTools
6) Test test test

If failure occurs, shut down everything and fall back to volumes backed up
in step 3.

Any thoughts or comments? For instance, should I be concerned about
emptying the DB2 active logs, so that they won't fill up during testing and
mount tapes, which in turn update entries in our tape management system?
(I need to read more about DB2 Log Suspend also, to see if we need to do
this. I'm hoping that shutting down DB2 in step 1 takes care of this.)

Many thanks in advance!

Stg

----------------------------------------------------------------------------
-----
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home
page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select
"Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org.
The IDUG List Admins can be reached at [login to unmask email] Find
out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

Steve Grimes

Re: FDR Backup - Recovery
(in response to cliff boley)
Thanks Cliff!

I backed away from the imagecopy/recover approach, (although I've used it
successfully several times for application tables) because the OBID's etc.
would change when the upgrade dropped/recreated the tables. I guess
restoring the catalog and directory should take care of this, although
we've never done this. Ditto for archiving/recovering the log & bsds. And
I'll have to read to figure out what the DB2 ICF catalog is. (Is that
what I think of as the MVS User Catalog?)

Stg

Steve,
You must have imagecopy and recover utilities.
What it looks like your trying to do is the same a setting up for a
disaster
recovery.
What about imagcopy you peopletools tablespaces using listdef then
imagecopy the catalog and directory. Next archive the logs/bsds. Export the
DB2 ICF catalog, then FDR all of your DB2 distribution libraries.
If your upgrade fails and you need to fall back you could use your disaster
recovery procedures
to recover.

How does this sound?
cliff:-)

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

Nic Honan

Re: FDR Backup - Recovery
(in response to Steve Grimes)
I'd strongly recommend geting in some 3rd party tools. I can't cope with
out them.

I use BMC's Change Manager to copy different layers of PeopleSoft to
another layer. I take Image Copies and use their recover plus feature
(essentially a DSN1COPY like process for the purposes of this discusssion).

I've also heard of sites using CA's RC Migrator.

Try posting this question on a dedicated PeopleSoft for DB2 on z/OS
listserv:
http://www.ca.com/products/db2/email_forum.htm#PeopleSoft

Hope this helps - Nic

On Mon, 13 Dec 2004 12:23:04 -0600, [login to unmask email] wrote:

>Thanks Cliff!
>
>I backed away from the imagecopy/recover approach, (although I've used it
>successfully several times for application tables) because the OBID's etc.
>would change when the upgrade dropped/recreated the tables. I guess
>restoring the catalog and directory should take care of this, although
>we've never done this. Ditto for archiving/recovering the log & bsds. And
>I'll have to read to figure out what the DB2 ICF catalog is. (Is that
>what I think of as the MVS User Catalog?)
>
>Stg
>
>Steve,
>You must have imagecopy and recover utilities.
>What it looks like your trying to do is the same a setting up for a
>disaster
>recovery.
>What about imagcopy you peopletools tablespaces using listdef then
>imagecopy the catalog and directory. Next archive the logs/bsds. Export
the
>DB2 ICF catalog, then FDR all of your DB2 distribution libraries.
>If your upgrade fails and you need to fall back you could use your
disaster
>recovery procedures
>to recover.
>
>How does this sound?
>cliff:-)
>
>--------------------------------------------------------------------------
-------
>Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and
home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page
select "Join or Leave the list". The IDUG DB2-L FAQ is at
http://www.idugdb2-l.org. The IDUG List Admins can be reached at DB2-L-
[login to unmask email] Find out the latest on IDUG conferences at
http://conferences.idug.org/index.cfm

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

Venkat Srinivasan

Re: FDR Backup - Recovery
(in response to Nic Honan)
Steve,
Why do you think a Tools upgrade will be that complicated?. Normally a
Tools upgrade is a cake walk if you do it through the upgrade assistant.

You are complicating the whole process by planning a mega recovery
preparation. It would take a phenomenal time to test the process in the
first place. You need to consider the Web, App and File servers and
Process scheduler servers etc too as they could also be affected by the
upgrade.
Such a mega recovery, if not tested will present sweet surprises.

You would need to include source libs, hfs files as they are also part of
PeopleTools.

If a upgrade step fails, debug why it failed and re-run the step. Disaster
during tools upgrade is rare although I had one. Never think about system
restore until you have a valid reason to do so.

Being in PS world for more than 5 years, I am really sorry to say that the
plan will not work(I assume you have never tested this procedure) nor is
even necessary at all. However each installation should have a tested DR
procedure and I dont dispute this at all.

You may be able to hire consultants who specialize in upgrades who can
upgrade all your databases over a weekend. Call me offline (609-707-2080)
if you need help in exploring this route.

Regards,
venkat

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

Steve Grimes

Re: FDR Backup - Recovery
(in response to Venkat Srinivasan)
Hello -- and thanks Venkat, for your reply. I share my response here so
that others may benefit, or weep. ;-)

The plot has thickened considerably since I first started looking at this.
(Btw and unfortunately, tools don't seem to be an option.) Also, my part
in this pagent is only DB2 backup/recovery.

We have upgraded two subsystems so far, one using Upgrade Assistant (and
working around failures) and one by manually running the scripts. Both
resulting systems have had problems -- but the problems were different.
The first problems stemed from our application tables not matching our
tools metadata. We're fixing this, but can't fix it in production before
the upgrade because we've already upgraded the development and test
systems! We can't transfer (I'm told) from the 8.45 development/test
systems to the 8.41 production system because we can't transfer between PT
versions.

There we're other problems too, that we haven't completely sorted through.
There were LDAP code customization problems. Other's had to do with some
security links lost between permission lists and content references.
Performance also seems to be "different". (One issue here is that we can't
let Peopletools do DSNUTIL Runstats when it wants to. We don't have
DSNUTIL's running -- I'm waiting on RRS for that. We think this should
only encumber App. Engines. We're running stats every night now, to
compensate for anywhere else they may be missed.)

The group doing the upgrade now has a 160 point checklist.

But behold, "With Restrict On Drop" should spare our application tables
from any untimely demise. I've developed a Rexx to add it to all (most) of
our production application tables. The FDR backups become insurance. I
have another Rexx that analysizes the subsystem and creates FDR jobs for
all volumes referenced (in either DB2 Stogroup definitions or by ICF
entries for the HLQ's we use.) I do hope we never have to use it.
Unfortunately, fallback to the FDR backups also means restoring at least
one ICF catalog (I'm downloading the redbook...)

We would still remain mostly "drained" during the entire upgrade (now
scheduled for MLK weekend) so that fallback to the FDR's would be possible.
We will also be "ghosting" all servers involved.

I think our 1st fallback strategy is to just recover the PeopleTools tables
(and Web/App servers.) Recalling that upgrade "drops" PeopleTool tables
(and tablespaces?), this basically invalidates our image copies (I think),
unless we a) recreate original tables with DDL that has Obid's, or b)
recreate tables and use an unload/load approach.

I'm supporting b). We are trying to get our hands around the current 8.41
DDL.

As for the source libs and hfs files. I'm not sure what you mean. I
don't think we're running any scripts that touch MF Pds's or libraries.
The upgrade on the USS side is stalled (and we think can be deferred
because we only run SQR's on Z/os -- no process scheduler.) The SQR's
continue to run fine -- even when we upgraded from MVS to Z/os, though none
of the Java stuff or other directories were brought forward! (Another
surprise for me.) We cannot initiate an FTP to the MF from off-platform,
so I don't think anything can be snuck in here that we wouldn't know about.
This means we would continue to run the SQR's under the old version.

I agree this should be easy. Maybe in the final analysis, it will be.
I'll pass on your suggestion about acquiring the help of an upgrade
consultant. The problem is, our situation is so idiosyncratic -- beginning
with not having a people tools application! Even the tech support at
Peoplesoft has told us that the Upgrade Assistant assumes you have one.
There was no "demo" database with our initial install. Everything we seem
to do with this seems "custom." (My knowledge here is 2nd hand.)

I agree with you and will be a strong advocate of the "fix it and go
forward" technique once we get started.

Stg

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

cliff boley

Re: FDR Backup - Recovery
(in response to Steve Grimes)
wooh, good luck dude.
cliff:-)

-----Original Message-----
From: [login to unmask email] [mailto:[login to unmask email]

Sent: Wednesday, December 15, 2004 12:43 PM
To: [login to unmask email]
Subject: Re: FDR Backup - Recovery


Hello -- and thanks Venkat, for your reply. I share my response here so
that others may benefit, or weep. ;-)

The plot has thickened considerably since I first started looking at this.
(Btw and unfortunately, tools don't seem to be an option.) Also, my part
in this pagent is only DB2 backup/recovery.

We have upgraded two subsystems so far, one using Upgrade Assistant (and
working around failures) and one by manually running the scripts. Both
resulting systems have had problems -- but the problems were different.
The first problems stemed from our application tables not matching our
tools metadata. We're fixing this, but can't fix it in production before
the upgrade because we've already upgraded the development and test
systems! We can't transfer (I'm told) from the 8.45 development/test
systems to the 8.41 production system because we can't transfer between PT
versions.

There we're other problems too, that we haven't completely sorted through.
There were LDAP code customization problems. Other's had to do with some
security links lost between permission lists and content references.
Performance also seems to be "different". (One issue here is that we can't
let Peopletools do DSNUTIL Runstats when it wants to. We don't have
DSNUTIL's running -- I'm waiting on RRS for that. We think this should
only encumber App. Engines. We're running stats every night now, to
compensate for anywhere else they may be missed.)

The group doing the upgrade now has a 160 point checklist.

But behold, "With Restrict On Drop" should spare our application tables
from any untimely demise. I've developed a Rexx to add it to all (most) of
our production application tables. The FDR backups become insurance. I
have another Rexx that analysizes the subsystem and creates FDR jobs for
all volumes referenced (in either DB2 Stogroup definitions or by ICF
entries for the HLQ's we use.) I do hope we never have to use it.
Unfortunately, fallback to the FDR backups also means restoring at least
one ICF catalog (I'm downloading the redbook...)

We would still remain mostly "drained" during the entire upgrade (now
scheduled for MLK weekend) so that fallback to the FDR's would be possible.
We will also be "ghosting" all servers involved.

I think our 1st fallback strategy is to just recover the PeopleTools tables
(and Web/App servers.) Recalling that upgrade "drops" PeopleTool tables
(and tablespaces?), this basically invalidates our image copies (I think),
unless we a) recreate original tables with DDL that has Obid's, or b)
recreate tables and use an unload/load approach.

I'm supporting b). We are trying to get our hands around the current 8.41
DDL.

As for the source libs and hfs files. I'm not sure what you mean. I
don't think we're running any scripts that touch MF Pds's or libraries.
The upgrade on the USS side is stalled (and we think can be deferred
because we only run SQR's on Z/os -- no process scheduler.) The SQR's
continue to run fine -- even when we upgraded from MVS to Z/os, though none
of the Java stuff or other directories were brought forward! (Another
surprise for me.) We cannot initiate an FTP to the MF from off-platform,
so I don't think anything can be snuck in here that we wouldn't know about.
This means we would continue to run the SQR's under the old version.

I agree this should be easy. Maybe in the final analysis, it will be.
I'll pass on your suggestion about acquiring the help of an upgrade
consultant. The problem is, our situation is so idiosyncratic -- beginning
with not having a people tools application! Even the tech support at
Peoplesoft has told us that the Upgrade Assistant assumes you have one.
There was no "demo" database with our initial install. Everything we seem
to do with this seems "custom." (My knowledge here is 2nd hand.)

I agree with you and will be a strong advocate of the "fix it and go
forward" technique once we get started.

Stg

----------------------------------------------------------------------------
-----
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home
page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select
"Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org.
The IDUG List Admins can be reached at [login to unmask email] Find
out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm

Venkat Srinivasan

Re: FDR Backup - Recovery
(in response to cliff boley)
Steve,
From what you are saying, I advise the following.
Recollect and Document what went wrong first before anyone forgets. You
failed but for a reason. Analyze the cause and determine how you will fix
it.
I am not sure but it is obvious that your team let the upgrade effort with
a novice that has little or no knowledge of PeopleSoft or the
components/architecure.

Obviously the team did not follow basic rules of engagement. You deserve a
money back if it was from a consulting organization.

You(Your team) have violated basic rules in maintaining a PS environment.
It is too late to talk about it now.
I would recommend a specialist that has PS / MVS expertise to assess the
situation before you proceed further.

What you want to do at this time?.
I would recommend re-establishing a PROD copy first. Whe you embark on
this, make sure you include all affected areas including the distributed
side, more specifically the file server. PS 8.4X and upwards do not
necessarily need a file server by definition due to the way the
installation works except for MVS users. This is a classic example. If you
dont have one, you will create one. Re-create DEMO instance. you should be
able to create it from the CD's.

PS service is not going to help in this case. They probably are not going
to turn you away from indirect support either(I am not advocating PS in
any manner). You would need to choose bringing in a PS upgrade specialist.
You would also need a MVS/DB2/ PS specialist. If you are the specialist,
then work with the specialist to define a plan.

I can offer you an hour of consulting and I dont want to charge any money
at all.

The basis will be to help you define the problem and also help you create
an identical environment close to PROD in all (functional) aspects.

You can then do as many upgrades to your hearts satisfaction as you want.
Failure to do this may put your organization at a higher risk of loosing
PROD completely.

Bottomline, don't rush and your mega recovery may infact fail(sorry, Mr
Murphy can come is at anytime even when you ave clearly defined DR
scenario). I dont know how critical the apps is. But again, you dont want
to jettison a PROD environment without any backup plans. Your backup plans
need to be accurate without a single point of failure.
One time when I lost an instance was when a REORG REUSE on an object
with "DEFINE NO" created problems in DB2 V7.
If you see my presentations, I would always caution against REUSE and I
still continue to do so even after all those bugs have been fixed. You
never know when things start falling apart and it doesnt hurt preparing
for that.

This is a rare case.

Regards,
Venkat

---------------------------------------------------------------------------------
Welcome to the IDUG DB2-L list. To unsubscribe, go to the archives and home page at http://www.idugdb2-l.org/archives/db2-l.html. From that page select "Join or Leave the list". The IDUG DB2-L FAQ is at http://www.idugdb2-l.org. The IDUG List Admins can be reached at [login to unmask email] Find out the latest on IDUG conferences at http://conferences.idug.org/index.cfm