Logs analysis

Bharath Nunepalli

Logs analysis

Hi All,

 

We don't use any Tool to read and analyze the SYSLOG or DB2 error logs. 

Just curious to know what Tools are available for logs analysis. Please share your experiences with such Tools.

 

Thanks.

Rob Barbour

Logs analysis
(in response to Bharath Nunepalli)
Hi Bharath,

Have a look at ESAi / UBS *ULT4DB2™*, the log tracker/analyzer for DB2
z/OS®. See PDF overview http://www.esaigroup.com/DB2Overview.pdf (click
product image in PDF for detail product sheet).

Contact me if you have any questions,
Rob Barbour, ESAi


On 4/18/2017 12:28 PM, Bharath Nunepalli wrote:
>
> Hi All,
>
> We don't use any Tool to read and analyze the SYSLOG or DB2 error logs.
>
> Just curious to know what Tools are available for logs analysis.
> Please share your experiences with such Tools.
>
> Thanks.
>
>
> -----End Original Message-----

--
Signature Best Regards,
Rob Barbour
Enterprise Systems Associates, Inc ("ESAi")
UCF Research Park
3259 Progress Drive
Orlando, Florida 32826 USA
Toll Free: 1-866-GO-4-ESAI (1-866-464-3724)
http://www.ESAIGroup.com/products
http://www.ESAIGroup.com/products
twitter.com/ESAiSoftware http://www.twitter.com/ESAiSoftware

Thanks to all who visited us.


z Performance... Productivity for DB2... Services...
BCV4 - DB2, SAP, PeopleSoft Clones/Refreshes in Minutes vs Days
BCV5/BCV6 - Save 90% in CPU & Clock Time for DB2 Refresh / Migrate
XDM - Test Data Mgmt & Masking for DB2,Oracle, SQL Server et.al.
ULT4DB2 - Faster, Better Value in DB2 Log Analyzers w/ PROP, & Audit
BPA4DB2 - Save $, Improve DB2 Performance with Buffer Pool Analyzer
XM4DB2 - Proactive Approach for DB2,Performance, & Dynamic SQL
SQLQC - Find, Analyze, Improve SQL Quality Control and Performance

zDYNACAP - Meet SLAs at Lowest MLC Cost with Capacity Balancing
INSPECT-CPU - Improve CICS Application Performance with ICPU





Mark Vickers

Logs analysis
(in response to Bharath Nunepalli)
We use BMC Log Master and very happy with it.


[Signature5]


From: Bharath Nunepalli [mailto:[login to unmask email]
Sent: Tuesday, April 18, 2017 11:29 AM
To: [login to unmask email]
Subject: [DB2-L] - Logs analysis


Hi All,



We don't use any Tool to read and analyze the SYSLOG or DB2 error logs.

Just curious to know what Tools are available for logs analysis. Please share your experiences with such Tools.



Thanks.

-----End Original Message-----
Attachments

  • image001.png (21.5k)

Rob Barbour

RE: Logs analysis
(in response to Bharath Nunepalli)

Hi Bharath,

Have a  look at ESAi / UBS  ULT4DB2™, the log tracker/analyzer for DB2 z/OS®.  See PDF overview http://www.esaigroup.com/DB2Overview.pdf   (click product image in PDF for detail product sheet).    

Contact me offline if you have any questions,
--
Best Regards,
Rob Barbour
Enterprise Systems Associates, Inc ("ESAi")
UCF Research Park
3259 Progress Drive
Orlando,  Florida  32826    USA  
Toll Free: 1-866-GO-4-ESAI (1-866-464-3724)
http://www.ESAIGroup.com/products

twitter.com/ESAiSoftware
z Performance... Productivity for DB2... Services...
BCV4 -  DB2, SAP, PeopleSoft Clones/Refreshes in Minutes vs Days
BCV5/BCV6 -  Save 90% in CPU & Clock Time for DB2 Refresh / Migrate
XDM - Test Data Mgmt & Masking for DB2,Oracle, SQL Server et.al.
ULT4DB2 - Faster, Better Value in DB2 Log Analyzers w/ PROP, & Audit
BPA4DB2 - Save $, Improve DB2 Performance with Buffer Pool Analyzer
XM4DB2 - Proactive Approach for DB2,Performance, & Dynamic SQL
SQLQC - Find, Analyze, Improve SQL Quality Control and Performance

zDYNACAP - Meet SLAs at Lowest MLC Cost with Capacity Balancing
INSPECT-CPU - Improve CICS Application Performance with ICPU


Venkat Srinivasan

RE: Logs analysis
(in response to Rob Barbour)

I don't think the question was for DB2 recovery log. He wants to analyze SYSLOG / Error log. While I don't know what error log the context refers, zAWARE looks at operlog which constitutes messages written to syslog and implicitly MSTR. 

If the context is recovery log then BMC, CA, IBM (perhaps rocket) all have tooling around that. There are others too.

Venkat  
 
In Reply to Rob Barbour:

Hi Bharath,

Have a  look at ESAi / UBS  ULT4DB2™, the log tracker/analyzer for DB2 z/OS®.  See PDF overview http://www.esaigroup.com/DB2Overview.pdf   (click product image in PDF for detail product sheet).    

Contact me offline if you have any questions,
--
Best Regards,
Rob Barbour
Enterprise Systems Associates, Inc ("ESAi")
UCF Research Park
3259 Progress Drive
Orlando,  Florida  32826    USA  
Toll Free: 1-866-GO-4-ESAI (1-866-464-3724)
http://www.ESAIGroup.com/products

twitter.com/ESAiSoftware
z Performance... Productivity for DB2... Services...
BCV4 -  DB2, SAP, PeopleSoft Clones/Refreshes in Minutes vs Days
BCV5/BCV6 -  Save 90% in CPU & Clock Time for DB2 Refresh / Migrate
XDM - Test Data Mgmt & Masking for DB2,Oracle, SQL Server et.al.
ULT4DB2 - Faster, Better Value in DB2 Log Analyzers w/ PROP, & Audit
BPA4DB2 - Save $, Improve DB2 Performance with Buffer Pool Analyzer
XM4DB2 - Proactive Approach for DB2,Performance, & Dynamic SQL
SQLQC - Find, Analyze, Improve SQL Quality Control and Performance

zDYNACAP - Meet SLAs at Lowest MLC Cost with Capacity Balancing
INSPECT-CPU - Improve CICS Application Performance with ICPU

Bharath Nunepalli

RE: Logs analysis
(in response to Venkat Srinivasan)

Venkat,

You are right. I'm not asking about analyzing DB2 recovery logs, but about SYSLOG/MSTR logs

Jørn Thyssen

RE: Logs analysis [AD]
(in response to Bharath Nunepalli)

Hi,

zAware (part of IBM Operations Analytics for z Systems) has already been mentioned.

If you want to roll your own solution you can use Spark as your analytics platform maybe combined with visualisation like Apache Zeppelin. IBM Platform for Apache Spark for z/OS includes a mainframe data  service component for accessing data like SYSLOG, OPERLOG, SMF and more. 

 

Edit: sorry, forgot [AD]

Best regards,

Jørn Thyssen

Works for IBM Denmark. Views are personal. 

Edited By:
Jørn Thyssen[Organization Members] @ Apr 18, 2017 - 10:27 PM (Europe/Copenhagen)

Javier Estrada

RE: Logs analysis [AD]
(in response to Jørn Thyssen)

Hi:

  We use the CA suite and we're fine with it.

BTW... I'm also a Spark-Python enthusiast on distributed platforms, and I've heard a hundred times about the advantages on z/OS but I've never seen anyone with the actual "redbook/manual/book/tutorial/paper/theory/idea" on how to get started reading the logs or SMF (or reading anything that is not inside USS). If by any chance anyone gets any of this and is willing to share it, I would be forever grateful, as the IBM guys in here were unable to help.

 

Regards,

Javier Estrada

Certified DB2 11 for z/OS System Administrator. Mexico

Lizette Koehler

Logs analysis [AD]
(in response to Javier Estrada)
Javier,


Start with the SMF (System Manage Facility) for reading and copying(offloading) SMF data.



You will need to know if you are using SMF or SMF Logger for this process.



Once you get the SMF records of choice, then you will need to identify a process to read them.



Suggested utilities are (depending on your needs):



DAF from cbttape.org

DFSORT ICETOOL

Syncsort SYNCTOOL

CA SMF Director

CA MICS/SAS

Merrill MXG/SAS

SAS ITRM



Roll your own via SAS, Cobol, etc.



Probably other processes as well.



As for LOGs if you are asking about SYSLOG, then REXX and the SYSLOG stream will be fine. You will need to find out whether or not the system team spins the syslog and if they do, where they locate it.



There is also SDSF REXX that could also be helpful. There is a REDBOOK on it.





Lizette



From: Javier Estrada [mailto:[login to unmask email]
Sent: Friday, April 21, 2017 10:02 AM
To: [login to unmask email]
Subject: [DB2-L] - RE: Logs analysis [AD]



Hi:

We use the CA suite and we're fine with it.

BTW... I'm also a Spark-Python enthusiast on distributed platforms, and I've heard a hundred times about the advantages on z/OS but I've never seen anyone with the actual "redbook/manual/book/tutorial/paper/theory/idea" on how to get started reading the logs or SMF (or reading anything that is not inside USS). If by any chance anyone gets any of this and is willing to share it, I would be forever grateful, as the IBM guys in here were unable to help.



Regards,

Javier Estrada

Certified DB2 11 for z/OS System Administrator. Mexico



Lizette Koehler

Logs analysis [AD]
(in response to Lizette Koehler)
I forgot. You can probably load the SMF data into a DB2 table for analysis as well.



Lizette





From: Lizette Koehler [mailto:[login to unmask email]
Sent: Friday, April 21, 2017 10:18 AM
To: [login to unmask email]
Subject: [DB2-L] - RE: Logs analysis [AD]



Javier,


Start with the SMF (System Manage Facility) for reading and copying(offloading) SMF data.



You will need to know if you are using SMF or SMF Logger for this process.



Once you get the SMF records of choice, then you will need to identify a process to read them.



Suggested utilities are (depending on your needs):



DAF from cbttape.org

DFSORT ICETOOL

Syncsort SYNCTOOL

CA SMF Director

CA MICS/SAS

Merrill MXG/SAS

SAS ITRM



Roll your own via SAS, Cobol, etc.



Probably other processes as well.



As for LOGs if you are asking about SYSLOG, then REXX and the SYSLOG stream will be fine. You will need to find out whether or not the system team spins the syslog and if they do, where they locate it.



There is also SDSF REXX that could also be helpful. There is a REDBOOK on it.





Lizette



From: Javier Estrada [mailto:[login to unmask email]
Sent: Friday, April 21, 2017 10:02 AM
To: [login to unmask email] <mailto:[login to unmask email]>
Subject: [DB2-L] - RE: Logs analysis [AD]



Hi:

We use the CA suite and we're fine with it.

BTW... I'm also a Spark-Python enthusiast on distributed platforms, and I've heard a hundred times about the advantages on z/OS but I've never seen anyone with the actual "redbook/manual/book/tutorial/paper/theory/idea" on how to get started reading the logs or SMF (or reading anything that is not inside USS). If by any chance anyone gets any of this and is willing to share it, I would be forever grateful, as the IBM guys in here were unable to help.



Regards,

Javier Estrada

Certified DB2 11 for z/OS System Administrator. Mexico





-----End Original Message-----

Bharath Nunepalli

RE: Logs analysis [AD]
(in response to Lizette Koehler)

I built a process using REXX that reads DB2 MSTR logs. I haven't included SYSLOG in that.

You can find more details in my blog https://bharathstechblog.blogspot.com/2016/09/db2-mstr-log-processing-db2-mstr-logs.html

 

 

Started this thread to find out whether there is any better product for logs analysis.

 

Roy Boxwell

Logs analysis [AD]
(in response to Lizette Koehler)
Are you crazy Lizette??? Why use a simple interface like DB2 when you can splunk it? Or transform across three systems...
I love Friday...

Roy Boxwell
SOFTWARE ENGINEERING GmbH and SEGUS Inc.
-Product Development-
Heinrichstrasse 83-85
40239 Düsseldorf/Germany
Tel. +49 (0)211 96149-675
Fax +49 (0)211 96149-32
Email: [login to unmask email]<mailto:[login to unmask email]>
http://www.seg.de

Software Engineering GmbH
Amtsgericht Düsseldorf, HRB 37894
Geschäftsführung: Gerhard Schubert, Bettina Schubert

On 21 Apr 2017, at 19:36, Lizette Koehler <[login to unmask email]<mailto:[login to unmask email]>> wrote:

I forgot. You can probably load the SMF data into a DB2 table for analysis as well.

Lizette


From: Lizette Koehler [mailto:[login to unmask email]
Sent: Friday, April 21, 2017 10:18 AM
To: [login to unmask email]<mailto:[login to unmask email]>
Subject: [DB2-L] - RE: Logs analysis [AD]

Javier,

Start with the SMF (System Manage Facility) for reading and copying(offloading) SMF data.

You will need to know if you are using SMF or SMF Logger for this process.

Once you get the SMF records of choice, then you will need to identify a process to read them.

Suggested utilities are (depending on your needs):

DAF from cbttape.org http://cbttape.org
DFSORT ICETOOL
Syncsort SYNCTOOL
CA SMF Director
CA MICS/SAS
Merrill MXG/SAS
SAS ITRM

Roll your own via SAS, Cobol, etc.

Probably other processes as well.

As for LOGs if you are asking about SYSLOG, then REXX and the SYSLOG stream will be fine. You will need to find out whether or not the system team spins the syslog and if they do, where they locate it.

There is also SDSF REXX that could also be helpful. There is a REDBOOK on it.


Lizette

From: Javier Estrada [mailto:[login to unmask email]
Sent: Friday, April 21, 2017 10:02 AM
To: [login to unmask email]<mailto:[login to unmask email]>
Subject: [DB2-L] - RE: Logs analysis [AD]


Hi:

We use the CA suite and we're fine with it.

BTW... I'm also a Spark-Python enthusiast on distributed platforms, and I've heard a hundred times about the advantages on z/OS but I've never seen anyone with the actual "redbook/manual/book/tutorial/paper/theory/idea" on how to get started reading the logs or SMF (or reading anything that is not inside USS). If by any chance anyone gets any of this and is willing to share it, I would be forever grateful, as the IBM guys in here were unable to help.



Regards,

Javier Estrada

Certified DB2 11 for z/OS System Administrator. Mexico


-----End Original Message-----

-----End Original Message-----

Javier Estrada

RE: Logs analysis [AD]
(in response to Roy Boxwell)

Hi:

  I like your blog, I read a few posts already.

Thank you all for your ideas as well :)

Err.... what I meant a few days ago is that yes, Spark on z/OS looks very promising only if it could be as friendly as it is with other platforms, last time I talked with one of the dev guys from IBM he told me "usually when a client wants to read the SMF with Spark, they have to make us go to their site, there's no official documentation for that and there's still no full support with Python as with the original Spark".

 

I also read the post on your REXX approach, that's actually the very same idea I would have worked with if I could extract the logs and them move them via NFS to an environment where I can work with a full Python/Anaconda/Canopy/anything-that-runs-Spak-and-Pandas-and-can-actually-create-visualizations.

 

Regards,

Javier Estrada

Certified DB2 11 for z/OS System Administrator. Mexico

Philip Sevetson

Logs analysis [AD]
(in response to Javier Estrada)
**please note my email address change**
Javier,

Who are you writing to? Your email went to the DB2-L quasi-public distribution list.

Philip Sevetson
Computer Systems Manager
5 Manhattan West (33rd St at 10th Ave)
New York, NY 10001-2632
212-857-1688 w
917-991-7052 c
212-857-1659 f
[cid:[login to unmask email]

From: Javier Estrada [mailto:[login to unmask email]
Sent: Monday, April 24, 2017 11:20 AM
To: [login to unmask email]
Subject: [DB2-L] - RE: Logs analysis [AD]


Hi:

I like your blog, I read a few posts already.

Thank you all for your ideas as well :)

Err.... what I meant a few days ago is that yes, Spark on z/OS looks very promising only if it could be as friendly as it is with other platforms, last time I talked with one of the dev guys from IBM he told me "usually when a client wants to read the SMF with Spark, they have to make us go to their site, there's no official documentation for that and there's still no full support with Python as with the original Spark".



I also read the post on your REXX approach, that's actually the very same idea I would have worked with if I could extract the logs and them move them via NFS to an environment where I can work with a full Python/Anaconda/Canopy/anything-that-runs-Spak-and-Pandas-and-can-actually-create-visualizations.



Regards,

Javier Estrada

Certified DB2 11 for z/OS System Administrator. Mexico

-----End Original Message-----
**This e-mail, including any attachments, may be confidential, privileged, or otherwise legally protected. It is intended only for the addressee. If you received this e-mail in error or from someone who was not authorized to send it to you, do not disseminate, copy, or otherwise use this e-mail or its attachments. Please notify the sender immediately by reply e-mail and delete the e-mail from your system.**
Attachments

  • image001.png (3.3k)

Patrick Bossman

RE: Logs analysis [AD]
(in response to Philip Sevetson)

Anyone using ADMIN_INFO_SYSLOG to read SYSLOG?  

Bharath Nunepalli

RE: Logs analysis [AD]
(in response to Patrick Bossman)

Didn't try ADMIN_INFO_SYSLOG yet.

Bharath Nunepalli

RE: Logs analysis [AD]
(in response to Javier Estrada)

Javier,

Thanks for looking into my blog.