New Message: Re: Index root slowdowns

[webmaster at userland.com]

A new message was posted:

Address: http://manila.userland.com/discuss/msgReader$2991

By: Lawrence Lee (lawrence at userland.com)

Can you try inserting a debug command into this script:

mainResponder.search.server.deIndexPage

to check which of the two deindexing methods it's using in this section of code:

if content == nil { //the text hasn't been saved, use brute force
 searchEngine.deIndexPage (url, @[indexPath].index)}
else { //smart de-indexing
 «local (wordCount = string.countFields (content, ' '))
 local (i);
 local (ct = 0);
 «for i = 1 to wordCount
 «local (oneWord = string.nthField (content, ' ', i))
 «
 «sys.systemTask ()
 «
 «if sizeOf (oneWord) > 0
 «local (letter = oneWord [1])
 «
 «oneWord = string.popTrailing (oneWord, 's')
 «local (adrLetter = @[indexPath].index.[letter])
 «if defined (adrLetter^)
 «local (adrWord = @adrLetter^.[oneWord])
 «if defined (adrWord^) //is this word in the index?
 «local (adrRef = @adrWord^.[url])
 «if defined (adrRef^) //is this page in the index under this word?
 «delete (adrRef)
 «if sizeOf (adrWord^) == 0
 «delete (adrWord)
 «if sizeOf (adrLetter^) == 0
 «delete (adrLetter)
 «
 «msg (oneWord)
 while sizeOf (content) > 0 { //PBS 07/18/00: don't use string.nthField, but get the leading word, then delete from the content string
 content = string.trimWhiteSpace (content);
 local (oneWord = string.nthField (content, ' ', 1));
 local (wordSize = sizeOf (oneWord));
 oneWord = string.dropNonAlphas (oneWord);
 oneWord = string.lower (oneWord);
 oneWord = string.trimWhiteSpace (oneWord);
 oneWord = string.popTrailing (oneWord, 's'); //pop off trailing s's
 if oneWord == "" {
 content = string.delete (content, 1, wordSize);
 continue};
 if sizeOf (oneWord) > 0 {
 local (letter = oneWord [1]);

 oneWord = string.popTrailing (oneWord, 's');
 local (adrLetter = @[indexPath].index.[letter]);
 if defined (adrLetter^) {
 local (adrWord = @adrLetter^.[oneWord]);
 if defined (adrWord^) { //is this word in the index?
 local (adrRef = @adrWord^.[url]);
 if defined (adrRef^) { //is this page in the index under this word?
 delete (adrRef)};
 if sizeOf (adrWord^) == 0 {
 delete (adrWord)};
 if sizeOf (adrLetter^) == 0 {
 delete (adrLetter)}}}};

 content = string.delete (content, 1, wordSize);

 «ct++
 «if ct > 500 //relax on occassion
 «thread.sleepFor (0)
 «ct = 0}}

Lawrence

This is a Manila site... http://manila.userland.com/.