New Message: Databases corrupting

[webmaster at userland.com]

A new message was posted:

Address: http://frontier.userland.com/discuss/msgReader$14084

By: Robert Cassidy (rmcassid at uci.edu)

I'm having an increasingly frequent problem with one of my root files corrupting. I have 5500 user accounts that get heavy read/write access - frequent surveys, data updates from an external source, etc. It's safe to say that in any given month at least half of the accounts will be written to, and at times over half will be written to in a given day when a large update occurs.

As the number of accounts has grown and the traffic to my site grown as well, so have the corruption problems. I need to get this completely resolved at this point.

I've moved my user accounts into their own root file, apart from any other frontier data. It's only 20MB, so i don't think that file size is an issue. I suspect that the problem happens when a substantial number of data appends take place with no intervening compaction. I also supect that the per minute mainresponder file saves collides with other write activities by users or the system itself. I'm pretty sure it takes close to a full minute to save the user accounts and during heavy periods a user account will be written to every minute. (That is, I can watch the file being saved and simultaneously written to. I'm not sure how that can be without corrupting it.) I know I can turn that off, but there is no 'save every n minutes' option to replace it. I'm happy to write my own callback but I certainly know less how to do this properly than the Userland staff...

It's unclear to me how I can block other threads from saving or writing to a file while I do a large data update - it's a variable I can conrol and isolate, and I'd like to do that. It's also unclear to me how I can ensure my membership file is compacted on a regular basis and how to ensure that any problems are flagged.

The only thing I can see is to develop a process to save off the file (compacting it) before and after a large update, trap for compaction errors, do a file rename of the live, uncompacted file as well as the proper compacted file, swapping them. Then quitting and restarting Frontier so it picks up the compacted file as the live one. Other ideas?

This is a Manila site.. http://manila.userland.com/.