Alaska Software Inc. - Chr(0) characters in Character Fields
Username: Password:
AuthorTopic: Chr(0) characters in Character Fields
Regan CawkwellChr(0) characters in Character Fields
on Thu, 06 Sep 2018 10:03:58 +0100
Hi guys.

Has anyone seen a problem where some characters fields in some records get written 
containing chr(0) characters.  We've only noticed this recently but it might have been in 
the databases for quite a while.

I think this is some sort of corruption that has been introduced but I have no idea where 
it came from and what caused it.

Regan
Rba
Regan CawkwellRe: Chr(0) characters in Character Fields
on Fri, 07 Sep 2018 14:42:23 +0100
On 06/09/2018 10:03, Regan Cawkwell wrote:
> Hi guys.
> 
> Has anyone seen a problem where some characters fields in some records get written 
> containing chr(0) characters.  We've only noticed this recently but it might have been in 
> the databases for quite a while.
> 
> I think this is some sort of corruption that has been introduced but I have no idea where 
> it came from and what caused it.
> 
> Regan
> Rba

And I suppose the other question is, should I be worried about it?

Regan
Rba
James LoughnerRe: Chr(0) characters in Character Fields
on Sat, 08 Sep 2018 12:40:47 -0400
CHR(0) is end of string character for c and c++ programs. Do the values 
come from or manipulated by outside programs? Should not be a problem 
unless the CHR(0) is in the middle and the value is passed to some 
outside process. Might cause a problem in memo fields unless you are 
using FOX DBF and binary memos.

Jim

On 09/07/2018 09:42 AM, Regan Cawkwell wrote:
> On 06/09/2018 10:03, Regan Cawkwell wrote:
>> Hi guys.
>>
>> Has anyone seen a problem where some characters fields in some records 
>> get written containing chr(0) characters.  We've only noticed this 
>> recently but it might have been in the databases for quite a while.
>>
>> I think this is some sort of corruption that has been introduced but I 
>> have no idea where it came from and what caused it.
>>
>> Regan
>> Rba
> 
> And I suppose the other question is, should I be worried about it?
>
Regan CawkwellRe: Chr(0) characters in Character Fields
on Mon, 10 Sep 2018 12:33:53 +0100
On 08/09/2018 17:40, James Loughner wrote:
> CHR(0) is end of string character for c and c++ programs. Do the values come from or 
> manipulated by outside programs? Should not be a problem unless the CHR(0) is in the 
> middle and the value is passed to some outside process. Might cause a problem in memo 
> fields unless you are using FOX DBF and binary memos.
> 
> Jim
> 

Thanks, Jim.

Yes, I am aware that chr(0) is used as a EOL for strings in external systems.  Oh, and we 
do not use memo fields so at least that's one less thing to worry about.

In general, it is possible that the values in some fields can come from external sources 
and be passed to external sources at some point.  But no external program is used to 
actually write data to our databases (as far as we know).

Some of the fields we are finding these characters in appear to not be ones that could 
come from an external source.

A recent example is that we had reports of invoices missing at a customers site.  In this 
case, it appears that whole records which should have been filled with invoice related 
data instead have empty fields (it's almost as if as append had worked but then the data 
was not written).  But when we looked closer, each 'empty' character field in the record 
was made up of chr(0)s (which I find extremely odd).  And these records are written by our 
application alone.

At the same time this same customer had 2 missing orders (records) on their file, but when 
looking into that, the data was in the database but the index did not include the records 
(so a reindex fixed it).  And this occurred an hour after a reindex had already been done.

We open our databases with all related indexes so any database update should update the 
indexes at the same time, but that hasn't happened here.  And there was no evidence of any 
actual errors occurring.

So we seem to have weird things happening to both databases and indexes, which may or not 
be related.

I was asking about the chr(0)s as I thought they might cause index corruption or at least 
problems when using the indexes.

Regan
Rba
James LoughnerRe: Chr(0) characters in Character Fields
on Tue, 11 Sep 2018 09:49:10 -0400
On 9/10/18 7:33 AM, Regan Cawkwell wrote:
> On 08/09/2018 17:40, James Loughner wrote:
>> CHR(0) is end of string character for c and c++ programs. Do the 
>> values come from or manipulated by outside programs? Should not be a 
>> problem unless the CHR(0) is in the middle and the value is passed to 
>> some outside process. Might cause a problem in memo fields unless you 
>> are using FOX DBF and binary memos.
>>
>> Jim
>>
> 
> Thanks, Jim.
> 
> Yes, I am aware that chr(0) is used as a EOL for strings in external 
> systems.  Oh, and we do not use memo fields so at least that's one less 
> thing to worry about.
> 
> In general, it is possible that the values in some fields can come from 
> external sources and be passed to external sources at some point.  But 
> no external program is used to actually write data to our databases (as 
> far as we know).
> 
> Some of the fields we are finding these characters in appear to not be 
> ones that could come from an external source.
> 
> A recent example is that we had reports of invoices missing at a 
> customers site.  In this case, it appears that whole records which 
> should have been filled with invoice related data instead have empty 
> fields (it's almost as if as append had worked but then the data was not 
> written).  But when we looked closer, each 'empty' character field in 
> the record was made up of chr(0)s (which I find extremely odd).  And 
> these records are written by our application alone.
> 
> At the same time this same customer had 2 missing orders (records) on 
> their file, but when looking into that, the data was in the database but 
> the index did not include the records (so a reindex fixed it).  And this 
> occurred an hour after a reindex had already been done.
> 
> We open our databases with all related indexes so any database update 
> should update the indexes at the same time, but that hasn't happened 
> here.  And there was no evidence of any actual errors occurring.
> 
> So we seem to have weird things happening to both databases and indexes, 
> which may or not be related.
> 
> I was asking about the chr(0)s as I thought they might cause index 
> corruption or at least problems when using the indexes.
> 


Maybe network issues. One bad NIC or switch can cause corruption on 
networks. In the past I have seen networks cause index and data 
corruption. Fix the network and most corruption goes away.

Jim
Otto TrappRe: Chr(0) characters in Character Fields
on Mon, 24 Sep 2018 17:26:29 +0200
Hi Regan,

Last week a customer signalled lost invoices and after I analyzed their 
database I found just the same issue you described. Some records were 
all Chr(0) bytes. Also some other records had Chr(0) in some fields 
within the actual record but the rest of the record was intact. I even 
found one record where a character field had its data "right padded" 
with Chr(0).

This seems very organized compared to "ordinary" dbf corruptions where 
real data is filled with "random" bytes stretching over record boundaries.

The database is on a shared mapped drive (Linux with Samba).

We know that they write the dbf files from a Harbo*r executable as well 
but previously I never have seen such damaged records.

Very odd!

Best Regards,
Otto
Regan CawkwellRe: Chr(0) characters in Character Fields
on Mon, 24 Sep 2018 17:39:15 +0100
On 24/09/2018 16:26, Otto Trapp wrote:
> Hi Regan,
> 
> Last week a customer signalled lost invoices and after I analyzed their database I found 
> just the same issue you described. Some records were all Chr(0) bytes. Also some other 
> records had Chr(0) in some fields within the actual record but the rest of the record was 
> intact. I even found one record where a character field had its data "right padded" with 
> Chr(0).
> 
> This seems very organized compared to "ordinary" dbf corruptions where real data is filled 
> with "random" bytes stretching over record boundaries.
> 
> The database is on a shared mapped drive (Linux with Samba).
> 
> We know that they write the dbf files from a Harbo*r executable as well but previously I 
> never have seen such damaged records.
> 
> Very odd!
> 
> Best Regards,
> Otto

Hi Otto

I still don't know what to make of this.

I am beginning to wonder about the DBF being modified externally.

Regan
Rba