Memo file is missing/invalid error behavior in VFP9

[Giancarlo Piccinato]

Hi everybody.

I'm porting an ancient FPW application to VFP9.
Right now, I'm working on a procedure which allows the user to import
previous versions data. The process works fine exception made for those
cases in which the original table has some memo file corruption problem. For
some reason (design, I suppose) VFP9 seems to be unable to process the
records of a table after it encounters one affected by some memo field
problem. This behavior differs from earlier VFP versions.
Now the question: is there any workaround to address this issue?

Thanks in advance

[Josh Assing]

Try

SET TABLEVALIDATE TO 0

Post by Giancarlo Piccinato
Hi everybody.
I'm porting an ancient FPW application to VFP9.
Right now, I'm working on a procedure which allows the user to import
previous versions data. The process works fine exception made for those
cases in which the original table has some memo file corruption problem. For
some reason (design, I suppose) VFP9 seems to be unable to process the
records of a table after it encounters one affected by some memo field
problem. This behavior differs from earlier VFP versions.
Now the question: is there any workaround to address this issue?
Thanks in advance

--- AntiSpam/harvest ---
Remove X's to send email to me.

[Giancarlo Piccinato]

Hi Josh.

It won't work.
I've already tried setting tablevalidate to 0 through 15 or so with no luck.
I've also tried to select the records into a cursor and then copy the
records to another table. Same result.
I'll repeat: this wasn't an issue in prior VFP versions.
If you want me to I can send you some sample data (~41K).
I'm still open to suggestions.
--
Giancarlo Piccinato
www.piccinato.org

Post by Josh Assing
Try
SET TABLEVALIDATE TO 0
On Sat, 6 May 2006 10:12:41 -0600, "Giancarlo Piccinato"

Post by Giancarlo Piccinato
Hi everybody.
I'm porting an ancient FPW application to VFP9.
Right now, I'm working on a procedure which allows the user to import
previous versions data. The process works fine exception made for those
cases in which the original table has some memo file corruption problem. For
some reason (design, I suppose) VFP9 seems to be unable to process the
records of a table after it encounters one affected by some memo field
problem. This behavior differs from earlier VFP versions.
Now the question: is there any workaround to address this issue?
Thanks in advance

--- AntiSpam/harvest ---
Remove X's to send email to me.

[Josh Assing]

Can you post this dbf/fpt pair? I haven't seen an instance where previous
versions could open it and 9 couldn't with tablevalidate=0

-j

Post by Giancarlo Piccinato
Hi Josh.
It won't work.
I've already tried setting tablevalidate to 0 through 15 or so with no luck.
I've also tried to select the records into a cursor and then copy the
records to another table. Same result.
I'll repeat: this wasn't an issue in prior VFP versions.
If you want me to I can send you some sample data (~41K).
I'm still open to suggestions.

--- AntiSpam/harvest ---
Remove X's to send email to me.

[Sergey Berezniker]

Hi Giancarlo,

You can use TRY...ENDTRY to handle records with corrupted Memo as shown
below.
BTW, the table Especie you posted has index corruption. You would want
to fix that first.

*-------------------------------------------------------------------
CLEAR
CLOSE DATABASES ALL

LOCAL oExp AS Exception


lcTable = "Especie"
lcNewTable = "NewEspecie"
ERASE (FORCEEXT(lcNewTAble, "*"))
USE (lcTable) EXCLUSIVE
COPY STRUCTURE CDX TO (lcNewTable)

SELECT Especie
SCAN

TRY
SCATTER MEMO NAME oRec
INSERT INTO (lcNewTable) FROM NAME oRec
CATCH TO oExp WHEN oExp.ErrorNo = 41
? RECNO(), "- corrupted Memo"
SCATTER NAME oRec
INSERT INTO (lcNewTable) FROM NAME oRec
ENDTRY

ENDSCAN
*-------------------------------------------------------------------
--
--sb--

MVP VFP

Josh,
this isn't a question of being able to open the table or not because I can.
What I can't do is importing the entire table.
In the example I'm posting (hope you all won't mind) I can import records up
to number 675.
Starting from record 676 through 680, VFP9 won't copy any data whereas
previous VFP versions do.
Thanks for your help.

[Giancarlo Piccinato]

Hello Sergey.

In your example you're omitting memo fields altogether. So, I guess this is
not the solution I was after because that would mean losing great deals of
data, albeit not in this particular case. Furthermore, I sometimes use the
export/import thing to retrieve data from corrupted tables and that kind of
approach would slow down the process considerably (I’m talking 1M+ records
here).

Post by Sergey Berezniker
BTW, the table Especie you posted has index corruption.

I didn't realize that. Here's something funny though: VFP8 can reindex the
table without a flinch, VFP9 cannot (stops at 115 records).
Anyways, that's not an issue because I always rebuild indices from scratch.

Thank you,

Giancarlo

Post by Sergey Berezniker
Hi Giancarlo,
You can use TRY...ENDTRY to handle records with corrupted Memo as shown
below.
BTW, the table Especie you posted has index corruption. You would want to
fix that first.
*-------------------------------------------------------------------
CLEAR
CLOSE DATABASES ALL
LOCAL oExp AS Exception
lcTable = "Especie"
lcNewTable = "NewEspecie"
ERASE (FORCEEXT(lcNewTAble, "*"))
USE (lcTable) EXCLUSIVE
COPY STRUCTURE CDX TO (lcNewTable)
SELECT Especie
SCAN
TRY
SCATTER MEMO NAME oRec
INSERT INTO (lcNewTable) FROM NAME oRec
CATCH TO oExp WHEN oExp.ErrorNo = 41
? RECNO(), "- corrupted Memo"
SCATTER NAME oRec
INSERT INTO (lcNewTable) FROM NAME oRec
ENDTRY
ENDSCAN
*-------------------------------------------------------------------
--
--sb--
MVP VFP

Josh,
this isn't a question of being able to open the table or not because I can.
What I can't do is importing the entire table.
In the example I'm posting (hope you all won't mind) I can import records up
to number 675.
Starting from record 676 through 680, VFP9 won't copy any data whereas
previous VFP versions do.
Thanks for your help.

[Sergey Berezniker]

Hi Giancarlo,

Post by Giancarlo Piccinato
Hello Sergey.
In your example you're omitting memo fields altogether. So, I guess this is
not the solution I was after because that would mean losing great deals of
data, albeit not in this particular case.

No, the code only skips memos for the records where memo is corrupted.

<snip>

[Giancarlo Piccinato]

Exactly. That's what I meant.
--
Giancarlo Piccinato
www.piccinato.org

Post by Sergey Berezniker
Hi Giancarlo,

Post by Giancarlo Piccinato
Hello Sergey.
In your example you're omitting memo fields altogether. So, I guess this is
not the solution I was after because that would mean losing great deals of
data, albeit not in this particular case.

No, the code only skips memos for the records where memo is corrupted.
<snip>
--
--sb--
VFP MVP

[Sergey Berezniker]

Post by Giancarlo Piccinato
Exactly. That's what I meant.

You can use the same idea (TRY...ENDTRY) to skip only corrupted memos
in case where're multiple memo fields per record.

[Giancarlo Piccinato]

Sergey,

I understand what you're saying, but I still think that the simplest
solution is sticking with VFP8.
I've already posted a request for change in behavior at the MS product
feedback site.
Something like SET MEMOVALIDATE 80 would be nice.

Thank you,
--
Giancarlo Piccinato
www.piccinato.org

Post by Giancarlo Piccinato
Exactly. That's what I meant.

You can use the same idea (TRY...ENDTRY) to skip only corrupted memos in
case where're multiple memo fields per record.
--
--sb--
VFP MVP

[Josh Assing]

I missed the database post -- can you email it to me?
josh at j assing dot com

if it's an index problem and your attempting to import it; delete the index; use
it w/o the cdx & import then?

Post by Giancarlo Piccinato
Hello Sergey.
In your example you're omitting memo fields altogether. So, I guess this is
not the solution I was after because that would mean losing great deals of
data, albeit not in this particular case. Furthermore, I sometimes use the
export/import thing to retrieve data from corrupted tables and that kind of
approach would slow down the process considerably (I’m talking 1M+ records
here).

Post by Sergey Berezniker
BTW, the table Especie you posted has index corruption.

I didn't realize that. Here's something funny though: VFP8 can reindex the
table without a flinch, VFP9 cannot (stops at 115 records).
Anyways, that's not an issue because I always rebuild indices from scratch.
Thank you,
Giancarlo

Post by Sergey Berezniker
Hi Giancarlo,
You can use TRY...ENDTRY to handle records with corrupted Memo as shown
below.
BTW, the table Especie you posted has index corruption. You would want to
fix that first.
*-------------------------------------------------------------------
CLEAR
CLOSE DATABASES ALL
LOCAL oExp AS Exception
lcTable = "Especie"
lcNewTable = "NewEspecie"
ERASE (FORCEEXT(lcNewTAble, "*"))
USE (lcTable) EXCLUSIVE
COPY STRUCTURE CDX TO (lcNewTable)
SELECT Especie
SCAN
TRY
SCATTER MEMO NAME oRec
INSERT INTO (lcNewTable) FROM NAME oRec
CATCH TO oExp WHEN oExp.ErrorNo = 41
? RECNO(), "- corrupted Memo"
SCATTER NAME oRec
INSERT INTO (lcNewTable) FROM NAME oRec
ENDTRY
ENDSCAN
*-------------------------------------------------------------------
--
--sb--
MVP VFP

Josh,
this isn't a question of being able to open the table or not because I can.
What I can't do is importing the entire table.
In the example I'm posting (hope you all won't mind) I can import records up
to number 675.
Starting from record 676 through 680, VFP9 won't copy any data whereas
previous VFP versions do.
Thanks for your help.

--- AntiSpam/harvest ---
Remove X's to send email to me.

[Giancarlo Piccinato]

Post by Josh Assing
I missed the database post -- can you email it to me?

Sure.

Post by Josh Assing
if it's an index problem and your attempting to import it; delete the index; use
it w/o the cdx & import then?

Same result.

Thanks,

Giancarlo

[Josh Assing]

Here's the text of the email I sent Giancarlo:

That is very very curious that in 8 it copies 745 records, but in 9 in only
copies 676

I don't think the index has any issues - as w/ or w/o it it's the same result
The memo file is definitely bad.

TABLE SCAN
Records scanned:       745
  found clear:               745
  found Deleted:           0
  found Corrupt:           0

MEMO SCAN
Duplicate Pointers:      0
Invalid Block Pointers:          5
Pointers Out Of Range:           0
Blocks Too Long:        0
Block end past EOF:   0

Total Memo Errors:     5

MEMO SAMPLING (valid memos only)
Valid Memos Sampled:           627
Minimum Memo Length:          2
Average Memo Length:           30
Maximum Memo Length:         66
Bytes Used:                24716
Bytes Slack:                16308
Bytes Wasted:             320

OTHER
Total Time:                 0:0:0 seconds


Attached is a repaired file -- your memo was bad -- I also updated your index so
that it won't give you an error on invalid key lengths.

--- AntiSpam/harvest ---
Remove X's to send email to me.

[Giancarlo Piccinato]

Thanks Josh but I don't think that repairing the table was the main point of
my OP rather than retrieve the data from corrupted ones under VFP9.
So, in conclusion:
1. VFP8 can easily recover from *minor* memo file corruption.
2. VFP9 can't

Do you agree with me?

Ciao,

Giancarlo

Post by Josh Assing
That is very very curious that in 8 it copies 745 records, but in 9 in only
copies 676
I don't think the index has any issues - as w/ or w/o it it's the same result
The memo file is definitely bad.
TABLE SCAN
Records scanned: 745
found clear: 745
found Deleted: 0
found Corrupt: 0
MEMO SCAN
Duplicate Pointers: 0
Invalid Block Pointers: 5
Pointers Out Of Range: 0
Blocks Too Long: 0
Block end past EOF: 0
Total Memo Errors: 5
MEMO SAMPLING (valid memos only)
Valid Memos Sampled: 627
Minimum Memo Length: 2
Average Memo Length: 30
Maximum Memo Length: 66
Bytes Used: 24716
Bytes Slack: 16308
Bytes Wasted: 320
OTHER
Total Time: 0:0:0 seconds
Attached is a repaired file -- your memo was bad -- I also updated your index so
that it won't give you an error on invalid key lengths.
--- AntiSpam/harvest ---
Remove X's to send email to me.

[Josh Assing]

Post by Giancarlo Piccinato
Thanks Josh but I don't think that repairing the table was the main point of
my OP rather than retrieve the data from corrupted ones under VFP9.
1. VFP8 can easily recover from *minor* memo file corruption.

"recover"? NO -- ignore, yes.

Post by Giancarlo Piccinato
2. VFP9 can't
Do you agree with me?
Ciao,
Giancarlo

Post by Josh Assing
That is very very curious that in 8 it copies 745 records, but in 9 in only
copies 676
I don't think the index has any issues - as w/ or w/o it it's the same result
The memo file is definitely bad.
TABLE SCAN
Records scanned: 745
found clear: 745
found Deleted: 0
found Corrupt: 0
MEMO SCAN
Duplicate Pointers: 0
Invalid Block Pointers: 5
Pointers Out Of Range: 0
Blocks Too Long: 0
Block end past EOF: 0
Total Memo Errors: 5
MEMO SAMPLING (valid memos only)
Valid Memos Sampled: 627
Minimum Memo Length: 2
Average Memo Length: 30
Maximum Memo Length: 66
Bytes Used: 24716
Bytes Slack: 16308
Bytes Wasted: 320
OTHER
Total Time: 0:0:0 seconds
Attached is a repaired file -- your memo was bad -- I also updated your index so
that it won't give you an error on invalid key lengths.
--- AntiSpam/harvest ---
Remove X's to send email to me.

--- AntiSpam/harvest ---
Remove X's to send email to me.

[Giancarlo Piccinato]

This post might be inappropriate. Click to display it.

[Giancarlo Piccinato]

BTW, which tool did you use to analyze the table?
Looks interesting.
--
Giancarlo Piccinato
www.piccinato.org

Post by Josh Assing
That is very very curious that in 8 it copies 745 records, but in 9 in only
copies 676
I don't think the index has any issues - as w/ or w/o it it's the same result
The memo file is definitely bad.
TABLE SCAN
Records scanned: 745
found clear: 745
found Deleted: 0
found Corrupt: 0
MEMO SCAN
Duplicate Pointers: 0
Invalid Block Pointers: 5
Pointers Out Of Range: 0
Blocks Too Long: 0
Block end past EOF: 0
Total Memo Errors: 5
MEMO SAMPLING (valid memos only)
Valid Memos Sampled: 627
Minimum Memo Length: 2
Average Memo Length: 30
Maximum Memo Length: 66
Bytes Used: 24716
Bytes Slack: 16308
Bytes Wasted: 320
OTHER
Total Time: 0:0:0 seconds
Attached is a repaired file -- your memo was bad -- I also updated your index so
that it won't give you an error on invalid key lengths.
--- AntiSpam/harvest ---
Remove X's to send email to me.

[Josh Assing]

Post by Sergey Berezniker
BTW, the table Especie you posted has index corruption. You would want
to fix that first.

One could argue he had data corruption.

The index was valid -- it was the data causing the error.

He had an index on
LEFT(UPPER(COESPEC),4)

Changing it to

LEFT(PADR(UPPER(COESPEC),10),4)

fixes the "index corruption" -- which again; was "invalid key length" which was
a data issue. If COESPEC is always at least 4 characters, then his original
index would not be an issue; but leaving it less than 4 it would.


--- AntiSpam/harvest ---
Remove X's to send email to me.

[Paul Lee]

Post by Giancarlo Piccinato
Hi everybody.
I'm porting an ancient FPW application to VFP9.
Right now, I'm working on a procedure which allows the user to import
previous versions data. The process works fine exception made for
those cases in which the original table has some memo file corruption
problem. For some reason (design, I suppose) VFP9 seems to be unable
to process the records of a table after it encounters one affected by
some memo field problem. This behavior differs from earlier VFP
versions. Now the question: is there any workaround to address this
issue? ....

VFP8/9 is fussier regarding any table corruption. This may be a good
thing to detect/fix errors before they get worse.

You could use a good error scanning/repair utility before you import
the tables.

-----------------------------------------------------------------
Paul Lee ........... Abri Technologies ......... http://abri.com/
'Recover' - top rated FoxPro file repair utility.
-----------------------------------------------------------------