Alaska Software Inc. - Lenght of memo FOXCDX
Username: Password:
AuthorTopic: Lenght of memo FOXCDX
CarlosLenght of memo FOXCDX
on Wed, 14 Dec 2016 10:40:38 +0000
Hi,

In de attached example, the lenght of a memo fiel was defined to 10 but the 
file structure reports 4 byte.
Apparently works fine but can this inconsistency afect big files?

Thanks,
Carlos


STRU.zip
Andreas Gehrs-Pahl
Re: Lenght of memo FOXCDX
on Wed, 14 Dec 2016 14:27:16 -0500
Carlos,

>In de attached example, the lenght of a memo fiel was defined to 10 but the 
>file structure reports 4 byte.
>Apparently works fine but can this inconsistency afect big files?

Memo Fields are always 4 bytes long in VFP. They are implemented as a binary 
representation of a number (0 through 0xFFFFFFFF/4,294,967,295), which is a 
pointer to a memo "block" structure. To represent such a number, 4 bytes (or 
32 bits) are needed. If the pointer is to be interpreted as a signed value, 
only 31 bits would be used, giving a maximum (positive) value of just 
(0x7FFFFFFF/2,147,483,647).

In non-VFP database files, this pointer is implemented as a simple string, 
consisting of 10 digits, similar to StrZero(nPointer, 10), which is enough 
to represent either 2,147,483,647 or 4,294,967,295. The maximum length of a 
memo field value is determined by the Memo "Block" size, which for VFP files 
is by default 64 bytes, but can be set to anything between 33 bytes and 64 
KB (or 65,535 bytes).

This means that by default a VFP FPT file can be up to 128 GB in size, while 
at a 64 KB block size, the file can be as large as 128 TB, which is also the 
theoretical maximum length of a single Memo Field value. In practice, the 
file length is apparently limited to 16 TB (according to the Xbase++ docs), 
even though I have no idea why, as the math doesn't support that number. It 
is probably OS-limited, though.

OTOH, DBT files always have a memo block length of 512 bytes. Even though 
the Xbase++ documentation states that the length of DBT files and Memo 
fields is "unlimited", this isn't quite the case. The maximum length of a 
DBT memo file is actually 2 GB (limited by internal file pointer length 
limitations), which means about 4.3 million memo field blocks can be used, 
either for one large 2 GB memo field or up to about 4.3 million individual 
memo fields that are each less than 512 bytes long.

Finally, the character used to identify fixed-length character fields in a 
(DBF) database header is always "C" and the one used for memo fields is 
always "M". There is an additional byte in the field structure of VFP and 
FoxPro2 files that indicates if one of those fields is treated as a binary 
field or not. This has been implemented in Xbase++ by using the "X" and "V" 
field types for DbStruct() and DbCreate(). In FoxPro2 mode, their is no "X" 
type, and the binary "V" type uses 4 bytes, while the "M" type still uses 10 
bytes. In VFP, both types of memo field ("M" and "V") use 4 bytes.

The used length of Memo, Date, Logical, and some other fields is actually 
hard-coded in the DBF/FOX database engines, and it doesn't matter at all 
what value you put in the structure array, as it is completely ignored by 
DbCreate() for those types of fields. You will therefore always see a length 
of 4 for all "V" and "M" memo fields in VFP databases, no matter what you 
specify in the structure array.

I also recommend to RTFM on database engines, specifically the FOXDBE, and 
to use Google to find more details on VFP database structures, headers, and 
field types, if you need additional information.

Hope that helps,

Andreas

Andreas Gehrs-Pahl
Absolute Software, LLC

phone: (989) 723-9927
email: Andreas@AbsoluteSoftwareLLC.com
web:   http://www.AbsoluteSoftwareLLC.com
[F]:   https://www.facebook.com/AbsoluteSoftwareLLC
CarlosRe: Lenght of memo FOXCDX
on Thu, 15 Dec 2016 12:32:56 +0000
Andreas,

Thanks for your detailed explanation.
I always used DBFNTX but want change due to nice features of FOXCDX.

So, it will be good if in Xbase++ documentation "or 10" be removed.

FOXDBE (DATA component)
Mapping of FoxPro field types in the Xbase++ DDL
      Memo (binary) M 4 or 10 V M



BTW the Time stamp format accepted in this FOXDBE field type
      Time stamp T 8 T C

"2016-12-31T23:59:59" is ... and about range values

We always learn with you.

Regards,
Carlos



"Andreas Gehrs-Pahl" escreveu na mensagem 
news:myw0qa69o8el.1rzekadsgfftb.dlg@40tude.net...

Carlos,

>In de attached example, the lenght of a memo fiel was defined to 10 but the
>file structure reports 4 byte.
>Apparently works fine but can this inconsistency afect big files?

Memo Fields are always 4 bytes long in VFP. They are implemented as a binary
representation of a number (0 through 0xFFFFFFFF/4,294,967,295), which is a
pointer to a memo "block" structure. To represent such a number, 4 bytes (or
32 bits) are needed. If the pointer is to be interpreted as a signed value,
only 31 bits would be used, giving a maximum (positive) value of just
(0x7FFFFFFF/2,147,483,647).

In non-VFP database files, this pointer is implemented as a simple string,
consisting of 10 digits, similar to StrZero(nPointer, 10), which is enough
to represent either 2,147,483,647 or 4,294,967,295. The maximum length of a
memo field value is determined by the Memo "Block" size, which for VFP files
is by default 64 bytes, but can be set to anything between 33 bytes and 64
KB (or 65,535 bytes).

This means that by default a VFP FPT file can be up to 128 GB in size, while
at a 64 KB block size, the file can be as large as 128 TB, which is also the
theoretical maximum length of a single Memo Field value. In practice, the
file length is apparently limited to 16 TB (according to the Xbase++ docs),
even though I have no idea why, as the math doesn't support that number. It
is probably OS-limited, though.

OTOH, DBT files always have a memo block length of 512 bytes. Even though
the Xbase++ documentation states that the length of DBT files and Memo
fields is "unlimited", this isn't quite the case. The maximum length of a
DBT memo file is actually 2 GB (limited by internal file pointer length
limitations), which means about 4.3 million memo field blocks can be used,
either for one large 2 GB memo field or up to about 4.3 million individual
memo fields that are each less than 512 bytes long.

Finally, the character used to identify fixed-length character fields in a
(DBF) database header is always "C" and the one used for memo fields is
always "M". There is an additional byte in the field structure of VFP and
FoxPro2 files that indicates if one of those fields is treated as a binary
field or not. This has been implemented in Xbase++ by using the "X" and "V"
field types for DbStruct() and DbCreate(). In FoxPro2 mode, their is no "X"
type, and the binary "V" type uses 4 bytes, while the "M" type still uses 10
bytes. In VFP, both types of memo field ("M" and "V") use 4 bytes.

The used length of Memo, Date, Logical, and some other fields is actually
hard-coded in the DBF/FOX database engines, and it doesn't matter at all
what value you put in the structure array, as it is completely ignored by
DbCreate() for those types of fields. You will therefore always see a length
of 4 for all "V" and "M" memo fields in VFP databases, no matter what you
specify in the structure array.

I also recommend to RTFM on database engines, specifically the FOXDBE, and
to use Google to find more details on VFP database structures, headers, and
field types, if you need additional information.

Hope that helps,

Andreas

Andreas Gehrs-Pahl
Absolute Software, LLC

phone: (989) 723-9927
email: Andreas@AbsoluteSoftwareLLC.com
web:   http://www.AbsoluteSoftwareLLC.com
[F]:   https://www.facebook.com/AbsoluteSoftwareLLC
Andreas Gehrs-Pahl
Re: Lenght of memo FOXCDX
on Thu, 15 Dec 2016 15:03:14 -0500
Carlos,

>Thanks for your detailed explanation.

You are welcome.

>So, it will be good if in Xbase++ documentation "or 10" be removed.

>FOXDBE (DATA component)
>Mapping of FoxPro field types in the Xbase++ DDL
>Memo (binary) M 4 or 10 V M

It could be possible that there is a mode where the binary Memo field is 
implemented as a 10-character field, rather than a 4-byte field. I can't 
tell for sure, so the docs might be correct in stating "4 or 10".

>BTW the Time stamp format accepted in this FOXDBE field type
>Time stamp T 8 T C
>"2016-12-31T23:59:59" is ... and about range values

The Date-Time "T" field is indeed 8 bytes long, and consists of two 32-bit 
(4 byte) numbers (in little-endian format), representing the Date and Time.

The Date is represented as a Julian Date number, where 1582-10-15 is 2299161 
and today (2016-12-15) is 2457738. Today would be represented as: 8A802500 
in the database, which is the little-endian representation of 0x25808A, 
which is 2457738 (decimal).

The second 4 bytes are the Time, represented as milliseconds since midnight. 
So 13:52:35 would be saved as (13 * 3600 + 52 * 60 + 35) * 1000, which is: 
49955000 (decimal) or 0x2FA40B8, which would be saved as: B840FA02 (in 
little-endian format).

The FOXDBE converts this to and from a 12-character string in the format: 
YYYYMMDDHHmmSS. No other characters, like "-", "T", or ":" are accepted or 
returned, though. This also means that, even though the database internally 
can handle milliseconds, the DBE will always truncate this to full seconds.

So, Field->TimeStamp := "20161215135235" would be saved in the database as 
the following eight bytes: 8A 80 25 00 B8 40 FA 02.

>We always learn with you.

I'm glad, as that is the main reason why I post in these newsgroups.

Andreas

Andreas Gehrs-Pahl
Absolute Software, LLC

phone: (989) 723-9927
email: Andreas@AbsoluteSoftwareLLC.com
web:   http://www.AbsoluteSoftwareLLC.com
[F]:   https://www.facebook.com/AbsoluteSoftwareLLC