Hi guys,

I'm writing some code that analyses WAA log files.  You may have seen my
Webstats routine which generates statistics from the WAA logs - for an
example of output, see here: 

  http://idep.org.uk/xbase/webstats.html

The code I'm working on at the moment goes much further.  It is also a lot
faster - I can process an entire years worth of logs in around 12 seconds.

Anyway, I'd like to test against a larger set of logs.  So, who thinks they
have the largest WAA logs, and do you have a years worth?  More
importantly, can I have them?  

Please don't send them to me - I don't want my mailserver clogged up - let
me know the average and total size and if I want yours, I'll ask for them.

The 'winner' gets a copy of the stats generated 

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

Ask me about my vow of silence!

Phil,

I don't think the question should be

a) who has the largest WAA LOGs

it rather should be

b) who has the fastest Xbase++ routine to analyze them

I'd join the competition on b), provided I get a common LOG file 

Regards,
--
Hannes

> 
> Hi guys,
> 
> I'm writing some code that analyses WAA log files.  You may have seen my
> Webstats routine which generates statistics from the WAA logs - for an
> example of output, see here:
> 
>   http://idep.org.uk/xbase/webstats.html
> 
> The code I'm working on at the moment goes much further.  It is also a lot
> faster - I can process an entire years worth of logs in around 12 seconds.
> 
> Anyway, I'd like to test against a larger set of logs.  So, who thinks they
> have the largest WAA logs, and do you have a years worth?  More
> importantly, can I have them?
> 
> Please don't send them to me - I don't want my mailserver clogged up - let
> me know the average and total size and if I want yours, I'll ask for them.
> 
> The 'winner' gets a copy of the stats generated 
> 
> Regards,
> --
> Phil Ide

Hannes,

>b) who has the fastest Xbase++ routine to analyze them
Nah, you would lose  Phil is cheating by using perl! 

Regards
Sander Elias

Sander,

>>b) who has the fastest Xbase++ routine to analyze them
> Nah, you would lose  Phil is cheating by using perl! 

Dang! The secret's out .

Let me see - one years worth of logs from a system which is only used by 5
people, approx 20.1MB of data.

1. Scan through all logs, create daily logs from monthly logs.
2. Scan through daily data, creating summary logs (one per day)
3. Scan through entire years worth of data to generate HTML chart
   showing:

   y-axis: each form-function listed seperately
   x-axis: number of hits, maximum, minimum, average and total duration
           (in seconds) to service each request per form-function.

Total time: 13 seconds.

Think you can compete? 

Oh btw, I am currently optimising the code - that was just a first attempt!

Once a WAA log has been scanned, fetching and analysing the data becomes
quite trivial.  Even though the chart generation scans the entire years
worth of data, it takes less than a second including output of the HTML.
This makes it suitable to generate statistics and analysis in real-time
even over long data periods.  That means you can set the script as a cgi
script and get stats and charts (which include the currently running log)
in real-time to show up-to-the-second data.

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

We now return to our regularly scheduled flame-throwing.

Phil,

> Once a WAA log has been scanned, fetching and analysing the data becomes
> quite trivial.  Even though the chart generation scans the entire years
> worth of data, it takes less than a second including output of the HTML.
> This makes it suitable to generate statistics and analysis in real-time
> even over long data periods.  That means you can set the script as a cgi
> script and get stats and charts (which include the currently running log)
> in real-time to show up-to-the-second data.

This is so impressing that I dare to ask: do you know of a good tutorial
for Regular Expressions

Hannes

Hannes,

>> Once a WAA log has been scanned, fetching and analysing the data becomes
>> quite trivial.  Even though the chart generation scans the entire years
>> worth of data, it takes less than a second including output of the HTML.
>> This makes it suitable to generate statistics and analysis in real-time
>> even over long data periods.  That means you can set the script as a cgi
>> script and get stats and charts (which include the currently running log)
>> in real-time to show up-to-the-second data.
> 
> This is so impressing that I dare to ask: do you know of a good tutorial
> for Regular Expressions

Yes.  Download and install ActiveState Perl and look at the documentation,
specifically the items marked 'perlre' and 'prelretut' in the menu.

Hint:  after installing perl, run PPM3 from the command-line and enter at
the prompt:

search Date-Calc

When you see the list of available packages, select the latest version of
Date::Calc and note the number in the list, then type:

install n

...where n is the number from the list.

This installs the date calculation routines which allow you to do clever
things with date values.

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

When the wind is southerly, I know a hawk from a handsaw.

Hannes,

Ok, I'm working on a principle that 3 seconds is an ok amount of time to
generate a web page, provided it is the sort of page that is not requested
with a high degree of frequency.  In other words, 3 seconds is not too long
to wait for a page such as a statistics dump.

I've optimised the log-analyser, which converts WAA monthly logs into daily
logs, and produces daily summary files.  Where it used to take 12 seconds
to work its way through a years worth of data, it now takes between 8-9
seconds.  This would usually be run as a scheduled task once daily or
monthly.  Further runs can be made against the current WAA log, and this
can happen any number of times without damaging the analysis.

The routine which rummages through the analysis and daily log files takes
about 0.6 seconds to rummage through 354 days worth of data (there are days
where no activity occurred in WAA - usually weekends) and generate the
graphs.

So, to get a real-time analysis for a 12 month period up to the latest
second - worst case is towards the end of the month - takes about 1.5
seconds.  With so much 'spare' time (another 1.5 seconds!) I am adding
additional statistical information that I thought would have to have been
done as a seperate task, providing more information in a single hit.  I
doubt that I'll reach the 2 seconds mark. 

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

We give nothing as willingly as our advice.

Phil,

> Ok, I'm working on a principle that 3 seconds is an ok amount of time to
> generate a web page,

it seems you're sending an answer every 3 seconds to me...

Gee, hold your breath 

Let me digest the RegEx and Perl stuff first, then I do understand how
you analyze the WAA logs

Thanks for pointing me into a good direction re. Perl and RegEx,

Hannes

Hi Phil;
What does the HTML chart look like?
Can you show us one?

Allen,

> What does the HTML chart look like?
> Can you show us one?

Yes.  Hold your breath - We're working on it 

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

Read the dox?!!? Yea, rite...

Allen,

> Yes.  Hold your breath - We're working on it 

Release breath - better?

Analysis is displayed in 3 levels:

1. Server Level - generic stats applying to the entire server
   (actually, WAA instance)
2. Package Level - stats applying to individual packages
   this gives a more meaningful breakdown
3. Function Level - stats for individual or groups of
   form-functions

Take a look at this for the Server Level stats:

  http://www.idep.org.uk/xbase/waarc

Hover over the graphs to get additional information.

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

Never put off till tomorrow what you can do next week.

FYI- the hover does not work in Mozilla

Jim

Phil Ide wrote:

> Allen,
> 
> 
>>Yes.  Hold your breath - We're working on it 
> 
> 
> Release breath - better?
> 
> Analysis is displayed in 3 levels:
> 
> 1. Server Level - generic stats applying to the entire server
>    (actually, WAA instance)
> 2. Package Level - stats applying to individual packages
>    this gives a more meaningful breakdown
> 3. Function Level - stats for individual or groups of
>    form-functions
> 
> Take a look at this for the Server Level stats:
> 
>   http://www.idep.org.uk/xbase/waarc
> 
> Hover over the graphs to get additional information.
> 
> Regards,

Thanks, Phil
It's interesting how you created the graphs ... they look good!

Allen,

> Thanks, Phil
> It's interesting how you created the graphs ... they look good!

Thanks for the compliment - I'm only copying what I've seen elsewhere
though.

Note that the page is only a sample to show how charts can look.  The
stats page generated by the analysis for the server-level stats contains
much more information.

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

A computer's attention span is as long as its power cord.

Jimmy,

> FYI- the hover does not work in Mozilla

That's because it has issues with w3c compliance.  Try it in IE - all it is
is the ALT attribute of an image.

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

The dead outnumber the living more than 30 to 1.

Jimmy,

> 
>> FYI- the hover does not work in Mozilla

Ok, try this one:

  http://www.xodc.org.uk/waarc

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

I think, therefore I'm single.

Phil,
 
>>> FYI- the hover does not work in Mozilla
>
>Ok, try this one:
>
>  http://www.xodc.org.uk/waarc
>
Ok, that you are fixing this kind of problems, Ok!
But talking to yourself about it? 

Regards
Sander Elias

That one works

Jim

Phil Ide wrote:
> Jimmy,
> 
> 
>>>FYI- the hover does not work in Mozilla
> 
> 
> Ok, try this one:
> 
>   http://www.xodc.org.uk/waarc
> 
> Regards,

Sander Elias wrote:
> 
> Hannes,
> 
> >b) who has the fastest Xbase++ routine to analyze them
> Nah, you would lose  Phil is cheating by using perl! 

Ok, Phil wins 

A good reason for me to dig deeper into Perl, RegEx and PCRE

Regards,
--
Hannes

Hannes,

> A good reason for me to dig deeper into Perl, RegEx and PCRE

Hehehe - well, you know about my XbPCRE library, which supplies a class to
interface to the Perl Compatible Regex Library.  However, were you aware
that I have managed to embed a perl processor into Xbase++?  With this, you
can do very complex regex-based expressions, some of which are incredible:

   do a regex match against a variable
  variable =~ /regex/flags

   do a search-and-replace
  variable =~ s/regex/replacement/flags

e.g.
   replace all occurences of E or O, global replace,
   case insensitive, where E or O is not at start or end of word
  $string =~ s/.+([eo]*).+/i/ig

   convert a string into an array by
   by dividing elements at a token
   example: split string at CRLF's
  @array = split('/\n/', string)


   split a comma-seperated list.
   get the first 3 elements as scalar variables,
   get the remaining elements as an array:
  if( $csv =~ /([^,]+),([^,]+),([^,]+),(.*)/){
     $el1 = $1;
     $el2 = $2;
     $el3 = $3;
     @rest =~ split( /,/, $4 );
  }

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

Friends, Romans, countrymen, lend me your taglines!

Phil,

I'm wondering why I have not received a response from you for my eMail
responding to you.

Did you receive my eMail?  Did you receive an error message from my ISP?

Wondering,
--
Hannes

Hannes,

> I'm wondering why I have not received a response from you for my eMail
> responding to you.
> 
> Did you receive my eMail?  Did you receive an error message from my ISP?

Yes, I recieved it - many thanks for your response.  I've been overloaded
at work, but I'll get a response back to you today.

Regards,

Phil Ide

*******************************************
*   Xbase++ FAQ, Libraries and Sources:   *
*   goto: http://www.idep.org.uk/xbase    *
* --------------------------------------- *
* www.xodc.org.uk - openSource Dev-Center *
*******************************************

Sysops; the undoing of Big Brother.