AUGE_OHR wrote:

> i do load 1.000.000 line ( = 82Kb)  into a String, no Problem.
> than i try SUBSTR() on that String, no Problem
> 
> when i try to AADD(aBig,cResult) my RAM increase until crash ...

As far as I can see you are just splitting the file into "lines" - so there
is no need to read the whole file into memory.

I have a file class (see attachment) that can handle this without using
that much memory, see the following sample that reads a CSV file and
removes CHR(0) from it.

FUNCTION CleanupCSV( cFilename )
LOCAL oFile, nPos, cLine

   oFile := TFile():New(cFilename)
   oFile:LineBreak := CHR(0x0A)
   oFile:Open()
   DO WHILE ! oFile:Eof

      nPos := oFile:position
      cLine := oFile:ReadLine()
      IF AT(CHR(0x00), cLine) > 0
         cLine := STRTRAN(cLine, CHR(0x00), " ")
         oFile:Goto(nPos)
         oFile:Write(cLine)
      ENDIF

   ENDDO

   oFile:Flush()
   oFile:Close()

RETURN NIL


regards
Thomas

Hi Thomas,

that is not the point. Jimmy postet a problem we diskussed in the german
forum. And I started that thread because that happend in my application and I
didn't know what's wrong with my code. 

The point is: Why does the application crash? The problem is not the big
array. The problem is not to add some strings into such a big array (if this
string filled hard coded e.g. cString := "this is a dummy string"). The
problem is, if you add a lot of elements to an array, filled with substrings
from the big string, than the handles rise extemely (I can see that in the
MemoryWhatcher), and the application crashes with a xppfatal.

No question that there are other ways to to the work. But: Why does this way
Jimmy showed end in a crash? What is the problem?

Jan


Thomas Braun <spam@software-braun.de> wrote in message
news:1e0ksz09ep598$.l7l5xrrniq9h.dlg@40tude.net...
>AUGE_OHR wrote:
>
>> i do load 1.000.000 line ( = 82Kb)  into a String, no Problem.
>> than i try SUBSTR() on that String, no Problem
>> 
>> when i try to AADD(aBig,cResult) my RAM increase until crash ...
>
>As far as I can see you are just splitting the file into "lines" - so there
>is no need to read the whole file into memory.
>
>I have a file class (see attachment) that can handle this without using
>that much memory, see the following sample that reads a CSV file and
>removes CHR(0) from it.
>
>FUNCTION CleanupCSV( cFilename )
>LOCAL oFile, nPos, cLine
>
>   oFile := TFile():New(cFilename)
>   oFile:LineBreak := CHR(0x0A)
>   oFile:Open()
>   DO WHILE ! oFile:Eof
>
>      nPos := oFile:position
>      cLine := oFile:ReadLine()
>      IF AT(CHR(0x00), cLine) > 0
>         cLine := STRTRAN(cLine, CHR(0x00), " ")
>         oFile:Goto(nPos)
>         oFile:Write(cLine)
>      ENDIF
>
>   ENDDO
>
>   oFile:Flush()
>   oFile:Close()
>
>RETURN NIL
>
>
>regards
>Thomas

Jan Escholt wrote:

> No question that there are other ways to to the work. But: Why does this way
> Jimmy showed end in a crash? What is the problem?

I would think either handles are lost because of a bug or the tight loop
does not give enough time to the the garbage collector to release
resources. Creating 1 Million+ memory object handles in such a short time
might be too much for the Xbase++ runtime.

In both cases, this is most likely nothing to be solved here but a case for
the Alaska support.

Regardless of this it would be interesting to find out if the "direct read"
method works or not... (I would suspect not, because the amount of MOM
handles will be almost the same)...

Thomas

hi,

thx all for Answer

This is a Demo only to show Problem with AADD(aBig,cLongString)
while in "real" i use "small" Block with 512byte without any Problem

> I would think either handles are lost because of a bug or the tight loop
> does not give enough time to the the garbage collector to release
> resources. Creating 1 Million+ memory object handles in such a short time
> might be too much for the Xbase++ runtime.

*** Code ***
PROCEDURE MAIN
LOCAL aArray  := {}
LOCAL cString := Replicate("A",78)+CRLF
LOCAL i

FOR i = 1 TO 2500000
   AAdd(aArray, cString)
NEXT

? LEN(aArray)
WAIT
RETURN
*** Eof ***

it is no Problem to do 2.500.000 AADD(), it is only if using "cLongString"

Not tested yet : i "think" it might work if "cLongString" < 64kb ?

greetings by OHR
Jimmy

Hi Jimmy,

Am 24.08.2010 17:24, schrieb AUGE_OHR:

> *** Code ***
> PROCEDURE MAIN
> LOCAL aArray  := {}
> LOCAL cString := Replicate("A",78)+CRLF
> LOCAL i
>
> FOR i = 1 TO 2500000
>     AAdd(aArray, cString)
> NEXT

Yeah, very very large arrays are possible with this piece of code.
However, since all elements in the array hold the same string at
the end, basically a single handle was requested to do the entire
stuff.

Xbase++ optimizes quite well in this area.

   Andreas Herdt
   Alaska Software

--------------------------------------------------------------------

Technical Support:      support@alaska-software.com

News Server:            news.alaska-software.com
Homepage:               http://www.alaska-software.com
WebKnowledgeBase:       http://www.alaska-software.com/kbase.shtm

Fax European Office:    +49 (0) 61 96 - 77 99 99 23
Fax US Office:          +1 (646) 218 1281
--------------------------------------------------------------------

Hi Thomas,

Am 24.08.2010 16:29, schrieb Thomas Braun:

> Jan Escholt wrote:

> I would think either handles are lost because of a bug or the tight loop
> does not give enough time to the the garbage collector to release
> resources. Creating 1 Million+ memory object handles in such a short time
> might be too much for the Xbase++ runtime.

No it isn't. The access to a handle is streight forward - very performant.
The GC does not have any problems with the initial sample in this thread.
As there is no garbage (everything is referenced in the array) the GC has
a great time for relaxing 

Most of the stuff allocated in the loop is referenced by some handles on
the stack - which number is constant.

   Andreas Herdt
   Alaska Software

--------------------------------------------------------------------

Technical Support:      support@alaska-software.com

News Server:            news.alaska-software.com
Homepage:               http://www.alaska-software.com
WebKnowledgeBase:       http://www.alaska-software.com/kbase.shtm

Fax European Office:    +49 (0) 61 96 - 77 99 99 23
Fax US Office:          +1 (646) 218 1281
--------------------------------------------------------------------

Andreas Herdt wrote:

> No it isn't. The access to a handle is streight forward - very performant.
> The GC does not have any problems with the initial sample in this thread.
> As there is no garbage (everything is referenced in the array) the GC has
> a great time for relaxing 
> 
> Most of the stuff allocated in the loop is referenced by some handles on
> the stack - which number is constant.

Andreas, thanks for the clarification.

Thomas

Jimmy:

We recommend you read the "Efficient String Operations in Xbase++" document.

ftp://ftp.alaska-software.com/documents/readtxt.pdf

Regards. Diego

Diego,

the example on page 3 is near to the example Jimmy postet. I just added 
the one line to change the lenght of cBuffer with in the DO-WHILE loop 
according to the Alaska-example. But again: The application quits with 
too many Memory Handles.

Jan


Am 24.08.2010 13:17, schrieb J.A. Diego Kerejeta:
> Jimmy:
>
> We recommend you read the "Efficient String Operations in Xbase++"
> document.
>
> ftp://ftp.alaska-software.com/documents/readtxt.pdf
>
> Regards. Diego
>

Jan,

> the example on page 3 is near to the example Jimmy postet. I just added
> the one line to change the lenght of cBuffer with in the DO-WHILE loop
> according to the Alaska-example. But again: The application quits with too
> many Memory Handles.

I have not had any problems to process the text file in the test I've done
this morning. The results are in the image of my previous message.

I replaced the function Zerlege(cBuffer) to this one:

FUNCTION TxtLine_5( cText )
***************************
LOCAL aLines, cLine, nStart, nEnd
LOCAL nCount, nSize, nStep

nStart := 1
nCount := 0
nSize := 100
nStep := 500
aLines := Array( nSize )

DO WHILE ( nEnd := At( CRLF, cText, nStart ) ) > 0
   IF ++ nCount > nSize
      ASize( aLines, nSize += nStep )
   ENDIF
   aLines[ nCount ] := SubStr( cText, nStart, nEnd-nStart )
   nStart := nEnd + 2
ENDDO

IF nStart < Len( cText )
   IF ++ nCount > nSize
      RETURN AAdd( aLines, SubStr( cText, nStart ) )
   ENDIF
   aLines[ nCount ] := SubStr( cText, nStart )
ENDIF
RETURN ASize( aLines, nCount )

Regards. Diego

Also changing the code to the example on page 8 doesn't solve the problem.

I also tried to use a Sleep() after every 1000 or 10000 lines to give 
time to the GC. From Sleep(0) to Sleep(200). No improvement.

Jan


Am 24.08.2010 13:17, schrieb J.A. Diego Kerejeta:
> Jimmy:
>
> We recommend you read the "Efficient String Operations in Xbase++"
> document.
>
> ftp://ftp.alaska-software.com/documents/readtxt.pdf
>
> Regards. Diego
>

Hi,

Basically the paper discusses how to deal with strings in an
efficient way. At the end this means that very few intermediate
strings are created during the process - as much as possible will
be done via offset values.

To the original problem this means:

* apporoximatly 80 MB is read to the process.
* After everything (except CRLF) is copied to elements of the
   array approximatly 160 MB is required. After the initial
   string looses all references its space (80MB) is ready for
   reuse.
   (In general 160 MB is no big deal for the Xbase++ runtime)
* The amount of max memory required can be reduced when not the
   substrings are held in the array, instead the offset positions
   can be collected.

(Almost) no memory is required when this is mixed with Thomas
approach where the data resides on drive. All information required
is read directly from data.

BUT: At the end an array is required with one element for each
substring - is it the string itself or its offset.

   Andreas Herdt
   Alaska Software

--------------------------------------------------------------------

Technical Support:      support@alaska-software.com

News Server:            news.alaska-software.com
Homepage:               http://www.alaska-software.com
WebKnowledgeBase:       http://www.alaska-software.com/kbase.shtm

Fax European Office:    +49 (0) 61 96 - 77 99 99 23
Fax US Office:          +1 (646) 218 1281
--------------------------------------------------------------------

Hi Jimmy,

This problem may be caused by the limit on the number of memory handles 
that Xbase sets. If you use more than 2.000.000 handles then the program 
crashes. This bugged us for some time until Alaska provided the solution 
by making this limit on the number of memory handles configurable at 
compile time. This feature is included in hotfix rollup #36.

Kind regards,
Jan Groenestein

AUGE_OHR schreef:
> hi,
>  
> i do load 1.000.000 line ( = 82Kb)  into a String, no Problem.
> than i try SUBSTR() on that String, no Problem
>  
> when i try to AADD(aBig,cResult) my RAM increase until crash ...
>  
> v1.9.355, XP Sp3, 2GB no "SwapDisk"
>  
> greetings by OHR
> Jimmy
>  
> *** Code ***
> #include "Fileio.ch"
>  
> #define CRLF   CHR(13)+CHR(10)
> #define MAKEERROR
>  
> PROCEDURE Main
> LOCAL cFile    := "BIGTEXT.TXT"
> LOCAL cBuffer  := ""
> LOCAL aBig     := {}
> LOCAL nStart
> LOCAL nStop
>  
> IF FILE( cFile )
> ELSE
> ?  TIME()
> ?  "creating File"
>    nStart   := SECONDS()
> ?  CreFile( cFile )                       82.000.000
> ?  nStop    := SECONDS()-nStart           Sec.
>    WAIT
> ENDIF
>  
> ?  TIME()
> ?  "reading  File"
>    nStart   := SECONDS()
>    cBuffer  := ReadFile( cFile )
> ?  nStop    := SECONDS()-nStart           Sec
> ?  LEN(cBuffer)                          
>    WAIT
>  
> ?  TIME()
> ?  "SUBSTR   File"
>    nStart   := SECONDS()
>    aBig     := Zerlege(cBuffer)
> ?  nStop    := SECONDS()-nStart           0.06 Sec
> ?  LEN(aBig)                              *78
>    WAIT
>  
> RETURN
>  
> FUNCTION Zerlege(cBuffer)
> LOCAL aBig  := {}
> LOCAL cText
> LOCAL nStart
> LOCAL nStop
>  
> ?  iMax := LEN(cBuffer)
>    nStart := 1
>  
>    DO WHILE .T.
>       nStop  := AT(CRLF,cBuffer,nStart)
>       IF nStop > 0
>          cText := SUBSTR(cBuffer,nStart,nStop)
>          nStart := nStop+2
>       
>     this will increase RAM until crash
>    
> #IFDEF MAKEERROR
>          AADD(aBig ,cText )
> #ENDIF
>    
>     output is slow .. we know
>    
> #IFDEF MAKESLOW
> ?        nStart
> #ENDIF
>    
>     but this is 100x times slower until BUG apprear
>    
> #IFDEF MAKESLOW
> ?        cText
> #ENDIF
>       ELSE
>          EXIT
>       ENDIF
>    ENDDO
>  
> RETURN aBig
>  
> FUNCTION CreFile( cFile )
> LOCAL nSource, cBuffer
> LOCAL nBytesSum      := 0
> LOCAL cLine
> LOCAL i
>  
>    IF ( nSource := FCreate( cFile, FC_NORMAL ) ) == -1
>       FClose( nSource )
>       ? "Datei kann nicht erstellt werden:", cFile
>       RETURN FError()
>    ENDIF
>  
>    cLine := Replicate("A", 78 )+CRLF
>    FOR i = 1 TO 1000000
>       nBytesSum += FWrite( nSource, cLine )
>    NEXT
>  
>    FClose( nSource )
>  
> RETURN nBytesSum
>  
> FUNCTION ReadFile( cFile )
> LOCAL nSource, cBuffer
> LOCAL nLaenge     := 0
>  
>    nSource := FOpen( cFile, FO_READ )
>    IF FError() <> 0
>       ? "Datei kann nicht geöffnet werden:",cFile
>       RETURN ""
>    ENDIF
>  
>    FSEEK (nSource,0,FS_SET)                        && be sure on Start
> ?  nLaenge := FSEEK (nSource,0,FS_END)      && Ende suchen
>    FSEEK (nSource,0,FS_SET)                        && back to Start
>  
>    cBuffer := Space( nLaenge )                    make big buffer
>    FRead( nSource, @cBuffer, nLaenge )     read hole file
>  
>    FClose( nSource )
>  
> RETURN cBuffer                                       82kb String
> *** Eof ***

Hi Jan,

ah, yes, that would help a lot. How do I increase the number of the 
handles with this fix?

But the basic question isn't answered yet - why do the handles increase 
if a substring is added to the array, but not if a hard coded string is 
added? What is the difference?

Jan


Am 24.08.2010 21:27, schrieb Jan Groenestein:
> Hi Jimmy,
>
> This problem may be caused by the limit on the number of memory handles
> that Xbase sets. If you use more than 2.000.000 handles then the program
> crashes. This bugged us for some time until Alaska provided the solution
> by making this limit on the number of memory handles configurable at
> compile time. This feature is included in hotfix rollup #36.
>
> Kind regards,
> Jan Groenestein
>
> AUGE_OHR schreef:
>> hi,
>>
>> i do load 1.000.000 line ( = 82Kb) into a String, no Problem.
>> than i try SUBSTR() on that String, no Problem
>>
>> when i try to AADD(aBig,cResult) my RAM increase until crash ...
>>
>> v1.9.355, XP Sp3, 2GB no "SwapDisk"
>>
>> greetings by OHR
>> Jimmy
>>
>> *** Code ***
>> #include "Fileio.ch"
>>
>> #define CRLF CHR(13)+CHR(10)
>> #define MAKEERROR
>>
>> PROCEDURE Main
>> LOCAL cFile := "BIGTEXT.TXT"
>> LOCAL cBuffer := ""
>> LOCAL aBig := {}
>> LOCAL nStart
>> LOCAL nStop
>>
>> IF FILE( cFile )
>> ELSE
>> ? TIME()
>> ? "creating File"
>> nStart := SECONDS()
>> ? CreFile( cFile )  82.000.000
>> ? nStop := SECONDS()-nStart  Sec.
>> WAIT
>> ENDIF
>>
>> ? TIME()
>> ? "reading File"
>> nStart := SECONDS()
>> cBuffer := ReadFile( cFile )
>> ? nStop := SECONDS()-nStart  Sec
>> ? LEN(cBuffer) 
>> WAIT
>>
>> ? TIME()
>> ? "SUBSTR File"
>> nStart := SECONDS()
>> aBig := Zerlege(cBuffer)
>> ? nStop := SECONDS()-nStart  0.06 Sec
>> ? LEN(aBig)  *78
>> WAIT
>>
>> RETURN
>>
>> FUNCTION Zerlege(cBuffer)
>> LOCAL aBig := {}
>> LOCAL cText
>> LOCAL nStart
>> LOCAL nStop
>>
>> ? iMax := LEN(cBuffer)
>> nStart := 1
>>
>> DO WHILE .T.
>> nStop := AT(CRLF,cBuffer,nStart)
>> IF nStop > 0
>> cText := SUBSTR(cBuffer,nStart,nStop)
>> nStart := nStop+2
>>   this will increase RAM until crash
>> 
>> #IFDEF MAKEERROR
>> AADD(aBig ,cText )
>> #ENDIF
>> 
>>  output is slow .. we know
>> 
>> #IFDEF MAKESLOW
>> ? nStart
>> #ENDIF
>> 
>>  but this is 100x times slower until BUG apprear
>> 
>> #IFDEF MAKESLOW
>> ? cText
>> #ENDIF
>> ELSE
>> EXIT
>> ENDIF
>> ENDDO
>>
>> RETURN aBig
>>
>> FUNCTION CreFile( cFile )
>> LOCAL nSource, cBuffer
>> LOCAL nBytesSum := 0
>> LOCAL cLine
>> LOCAL i
>>
>> IF ( nSource := FCreate( cFile, FC_NORMAL ) ) == -1
>> FClose( nSource )
>> ? "Datei kann nicht erstellt werden:", cFile
>> RETURN FError()
>> ENDIF
>>
>> cLine := Replicate("A", 78 )+CRLF
>> FOR i = 1 TO 1000000
>> nBytesSum += FWrite( nSource, cLine )
>> NEXT
>>
>> FClose( nSource )
>>
>> RETURN nBytesSum
>>
>> FUNCTION ReadFile( cFile )
>> LOCAL nSource, cBuffer
>> LOCAL nLaenge := 0
>>
>> nSource := FOpen( cFile, FO_READ )
>> IF FError() <> 0
>> ? "Datei kann nicht geöffnet werden:",cFile
>> RETURN ""
>> ENDIF
>>
>> FSEEK (nSource,0,FS_SET) && be sure on Start
>> ? nLaenge := FSEEK (nSource,0,FS_END) && Ende suchen
>> FSEEK (nSource,0,FS_SET) && back to Start
>>
>> cBuffer := Space( nLaenge )  make big buffer
>> FRead( nSource, @cBuffer, nLaenge )  read hole file
>>
>> FClose( nSource )
>>
>> RETURN cBuffer  82kb String
>> *** Eof ***

hi,

> This problem may be caused by the limit on the number of memory handles 
> that Xbase sets. If you use more than 2.000.000 handles then the program

Question : what exact is "memory handles" ?

the "Handles" show by Taskmanager do not "jump" ... just "some" more,
but my (avaiable) RAM goes down.

also in 2nd Demo it is no Problem to AADD() 2.500.000x

ONLY when used SUBSTR(cLongString_64kb_and_more_kb)
you can "see" that Effect

> This feature is included in hotfix rollup #36.

which "Hotfix" do you meen ?
*** Qoute ***Hotfix-Rollup #36
FIX 109-6130 Division by zero wrong on Windows 64 Bit
FIX 109-6235 Number of memory objects not configurable
FIX 109-6250 DllUnload requires a timeframe of 300 milliseconds
FIX 109-6286 On 64 Bit operating systems division by zero not correct
FIX 109-6290 Wrong parameters in application started in debugger (Win 7)
FIX 109-6295 Error "Invalid numeric value" after call to COM/AcitveX
*** eof ***





>I have not had any problems to process the text file in the test I've done
>this morning. The results are in the image of my previous message.

@Diego : how much RAM do you have ? 32bit or 64bit ?
did you have a look into Taskmanager ?

my Demo Filesize is only 82kb.  i often have File with > 1 GB
and it work with "small" Block with each Line.

Jan try to read hole File into String (no Problem), SUBSTR()
which work fine too and than AADD(aBig, cResult) and now
the BUG (?) happend.

Did Xbase++ still have a 64kb "Limit" ... ?

Your "modifikation" look good, i have to test it

greetings by OHR
Jimmy

AUGE_OHR wrote:

> which "Hotfix" do you meen ?
[...]
> FIX 109-6235 Number of memory objects not configurable

Most likely the one quoted above...

Thomas

immie,

It is easy to test the memory handles configuration.
Just add to the projects .ARC file :
===================== snip ========================
#include "cdf.ch"

VERSION
  CDF_MOM_HANDLESPACE_ID  = "8000000"
===================== snap ========================
This raises the available number of memory handles to 8.000.000
You can trace the used- and available memory handles im MEMWATCH.

Regards,
Jan


AUGE_OHR schreef:
> hi,
> 
>> This problem may be caused by the limit on the number of memory handles 
>> that Xbase sets. If you use more than 2.000.000 handles then the program
> 
> Question : what exact is "memory handles" ?
> 
> the "Handles" show by Taskmanager do not "jump" ... just "some" more,
> but my (avaiable) RAM goes down.
> 
> also in 2nd Demo it is no Problem to AADD() 2.500.000x
> 
> ONLY when used SUBSTR(cLongString_64kb_and_more_kb)
> you can "see" that Effect
> 
>> This feature is included in hotfix rollup #36.
> 
> which "Hotfix" do you meen ?
> *** Qoute ***Hotfix-Rollup #36
> FIX 109-6130 Division by zero wrong on Windows 64 Bit
> FIX 109-6235 Number of memory objects not configurable
> FIX 109-6250 DllUnload requires a timeframe of 300 milliseconds
> FIX 109-6286 On 64 Bit operating systems division by zero not correct
> FIX 109-6290 Wrong parameters in application started in debugger (Win 7)
> FIX 109-6295 Error "Invalid numeric value" after call to COM/AcitveX
> *** eof ***
> 
> 
> 
> 
> 
>> I have not had any problems to process the text file in the test I've done
>> this morning. The results are in the image of my previous message.
> 
> @Diego : how much RAM do you have ? 32bit or 64bit ?
> did you have a look into Taskmanager ?
> 
> my Demo Filesize is only 82kb.  i often have File with > 1 GB
> and it work with "small" Block with each Line.
> 
> Jan try to read hole File into String (no Problem), SUBSTR()
> which work fine too and than AADD(aBig, cResult) and now
> the BUG (?) happend.
> 
> Did Xbase++ still have a 64kb "Limit" ... ?
> 
> Your "modifikation" look good, i have to test it
> 
> greetings by OHR
> Jimmy
> 
>

Jan,

where do I find the cdf.ch?

Jan



Am 25.08.2010 18:08, schrieb Jan Groenestein:
> immie,
>
> It is easy to test the memory handles configuration.
> Just add to the projects .ARC file :
> ===================== snip ========================
> #include "cdf.ch"
>
> VERSION
> CDF_MOM_HANDLESPACE_ID = "8000000"
> ===================== snap ========================
> This raises the available number of memory handles to 8.000.000
> You can trace the used- and available memory handles im MEMWATCH.
>
> Regards,
> Jan
>
>
> AUGE_OHR schreef:
>> hi,
>>
>>> This problem may be caused by the limit on the number of memory
>>> handles that Xbase sets. If you use more than 2.000.000 handles then
>>> the program
>>
>> Question : what exact is "memory handles" ?
>>
>> the "Handles" show by Taskmanager do not "jump" ... just "some" more,
>> but my (avaiable) RAM goes down.
>>
>> also in 2nd Demo it is no Problem to AADD() 2.500.000x
>>
>> ONLY when used SUBSTR(cLongString_64kb_and_more_kb)
>> you can "see" that Effect
>>
>>> This feature is included in hotfix rollup #36.
>>
>> which "Hotfix" do you meen ?
>> *** Qoute ***Hotfix-Rollup #36
>> FIX 109-6130 Division by zero wrong on Windows 64 Bit
>> FIX 109-6235 Number of memory objects not configurable
>> FIX 109-6250 DllUnload requires a timeframe of 300 milliseconds
>> FIX 109-6286 On 64 Bit operating systems division by zero not correct
>> FIX 109-6290 Wrong parameters in application started in debugger (Win 7)
>> FIX 109-6295 Error "Invalid numeric value" after call to COM/AcitveX
>> *** eof ***
>>
>>
>>
>>
>>
>>> I have not had any problems to process the text file in the test I've
>>> done
>>> this morning. The results are in the image of my previous message.
>>
>> @Diego : how much RAM do you have ? 32bit or 64bit ?
>> did you have a look into Taskmanager ?
>>
>> my Demo Filesize is only 82kb. i often have File with > 1 GB
>> and it work with "small" Block with each Line.
>>
>> Jan try to read hole File into String (no Problem), SUBSTR()
>> which work fine too and than AADD(aBig, cResult) and now
>> the BUG (?) happend.
>>
>> Did Xbase++ still have a 64kb "Limit" ... ?
>>
>> Your "modifikation" look good, i have to test it
>>
>> greetings by OHR
>> Jimmy
>>
>>

Jimmy, et al,

The problem is with SubStr() in combination with assigning the result of 
a SubStr() operation to an array. Also, you wrote several times in this 
thread that the file/string is "just 82kB" -- it is actually 8 MB! And 
because of the "defect" in SubStr(), Xbase++ will assign a copy of the 
entire 8MB string to each Array Item -- instead of just 79 characters -- 
using up all (accessible) physical memory in a very short time -- not 
just Memory Handles -- leading to the crash.

The simplest workaround -- until Alaska fixes this -- is to use a small 
replacement function for SubStr(), as shown below. Also, when dealing 
with (very) large strings, it is always a good idea to call sub-routines
with the "@" (by reference operator) for the large strings, particularly
if the called function doesn't actually change the original string. Your
code also added the CR (but not he LF) to the sub string to be added to 
the Array, which was probably unintentionally.

So, the attached, slightly modified program fixes those problems, but 
you still should report this to Alaska Support directly to get a PDR, 
as they (usually) won't create a PDR, if you just post something here.

-- Andreas

---                                                                      ---
  Andreas Gehrs-Pahl              E-Mail: GPahl@CharterMI.net
  415 Gute Street                     or: Andreas@DDPSoftware.com
  Owosso, MI 48867-4410               or: Andreas_Gehrs-Pahl@CrimeCog.com
  Tel: (989) 723-9927           Web Site: http://www.Aerospace-History.net
---                                                                      ---

Hi,

The issue solved in the meantime via our support channel.
Let me post our findings on this:

Am 24.08.2010 05:00, schrieb AUGE_OHR:
 > FUNCTION Zerlege(cBuffer)

(...)

 > DO WHILE .T.
 > nStop := AT(CRLF,cBuffer,nStart)
 > IF nStop > 0
 > cText := SUBSTR(cBuffer,nStart,nStop)

This one is an implemantation error in this sample code.
The third parameter is is not the "last position" it is
the number of characters to copy to the new string.

Because of the coding error above the entire memory of
the process will be occupied after several thousand
iterations - which is no problem regarding memory handles.

SubStr() returns one string - of course when this string
is assigned to an array elment then one memory handle is
required to reference it.

After the bug was fixed the sample code went through.

After the additional measurements to tackle PDR 6235 have
been applied there was no problem to deal with arrays of
2.500.000 elements in size.

 > nStart := nStop+2
 > 
 >  this will increase RAM until crash
 > 
 > #IFDEF MAKEERROR
 > AADD(aBig ,cText )
 > #ENDIF
 > 
 >  output is slow .. we know
 > 
 > #IFDEF MAKESLOW
 > ? nStart
 > #ENDIF

Output in an inner loop is always slow when done into
a CRT or console window. The main issue here is that
Xbase++ does automatic thread synchronization, thus
output is correct even if more then one thread is
writing to the same window.

For sample code that depends on fast tracing, please
consider to include "simpleio.ch", link VIO and overload
procedure APPSys(). No SAY/GET is possible, furthermore
things like "Wait" will not work but the output is
done to StdOut. The output then can be redirected
to a file - pretty fast. Please refer to simpleio.ch for
further details.

 > 
 >  but this is 100x times slower until BUG apprear
 > 
 > #IFDEF MAKESLOW
 > ? cText
 > #ENDIF
 > ELSE


Streaming 4 GB output needs some time. See the discussion
about the codeing error above.


Hope this helps.

   Andreas Herdt
   Alaska Software

--------------------------------------------------------------------

Technical Support:      support@alaska-software.com

News Server:            news.alaska-software.com
Homepage:               http://www.alaska-software.com
WebKnowledgeBase:       http://www.alaska-software.com/kbase.shtm

Fax European Office:    +49 (0) 61 96 - 77 99 99 23
Fax US Office:          +1 (646) 218 1281
--------------------------------------------------------------------