Database corruption - bad design

[visionik]

First off, let me say Vitamin D has been a great product. I've been testing it for a few weeks for potential use in seven installations with seven copies of the Pro installation. This issue has put me on the fence on the buying decision, so I'm hoping to find some form of resolution:

I just fell victim to this:

http://www.vitamindinc.com/support/ref_ ... ps_42.html

"Error message says, "Your database is corrupt, Vitamin D must close now"

In the rare but exceedingly unfortunate event that the database is corrupted due to power loss, event history is lost because it is stored in the database. The video files then get cleaned up because they are no longer in the database, but if you copy them out before starting up again, you may be able to recover the video."

I had a power outage at my facility. The design of how Vitamin D responds to this situation is, if I may be honest, a supremely bad one.

1. At a bare minimum you should strongly advise your customers that Vitamin D systems need to be installed on a UPS, otherwise you will may lose 100% of your video in a power out situation.

2. At a minimum, the dialog that says "Your database is corrupt, Vitamin D must close now" should at least say "Your database is corrupt, Vitamin D must close now. PLEASE NOTE that upon restarting Vitamin D all your video files will be deleted. If you wish to save them, back them up before restarting Vitamin D" or something similar.

3. More ideally, in the dialog above you should give an option to backup the database. This could be as simple as renaming the top-folder vitamin D video file "archive" folder "archive-backup", then creating a new "archive" folder.

4. Yet more ideally, you should deal with "database corruption" in a better way.

- Consider writing a seperate transactional text log file of events, that can be re-imported if the database becomes corrupt.

- Offer an option to backup the database file every 24 hours, or a user specified number of hours.

5. Most ideally, you should consider a different database engine that is not as sensitive to power-off corruption.

I have a few follow up questions:

1. Where are the database file(s) located, so I may set up a my own automated backup of the file.

2. What database engine are you using to store your data?

3. Do you have any other suggestions on how to prevent this situation or manage it better?

Thank you again for creating an otherwise great product.

[visionik]

After looking around a bit, it appears the database files are stored in the following location on a Mac:

/Users/(username)/Library/Application Support/Vitamin D Video

and include the following Sqlite files:

clipdb
clipdb-journal
objdb
objdb-journal
responsedb
responsedb-journal

If I

1. Stop Vitamin D
2. Copy those files
3. Restart Vitamin D

is that a fair backup strategy for this database?

BTW, I've explored the Vitamin D Sqlite files in the MesaSQLite viewer/editor. Looks interesting.

[JPElectron]

Any database is susceptible to these issues if it is shutoff dirty...

The design of your video camera system, including the Vitamin D Video server, without using a UPS (and one configured for graceful shutdown prior to the battery being depleted) is, if I may be honest, a supremely bad one.

[visionik]

Are you a Vitamin D employee? In any case my questions and comments above still need to be addressed.

I have been running a competitor to Vitamin D on my Mac Mini for three years now. It has not been on a UPS. Power has gone out probably 10 times in those three years. In those three years I have had not a single instance of database corruption. In fact I'm still running it, side by side with Vitamin D. It's awesome software but the motion detection... It's bad. Oy vey, am I sick of getting recordings of tree shadows when the Florida wind blows, and then not getting a recording of a person walking right up our driveway. So I am/was hoping Vitamin D can solve my issues.

I have another Mac Mini running my home automation software (Indigo) and home mail server (Communigate). Same situation. No UPS, 3+ years, 10+ power outages, no corruption.

I agree with you that systems that require a UPS should have a UPS, however not all systems in a home need a UPS.

I have been trying Vitamin D for about a month now. The *first* time the power went out the database corrupted.

I don't see anywhere in the Vitamin D documentation that makes it clear that Vitamin D is *that* sensitive to power outages and corrupt databases. A good design would make that clear.

Nothing in the dialog about database corruption makes it clear that you are going to lose 100% of your recording when you restart Vitamin D. A good design would make that clear.

While I will argue the top two points with all of m conviction, I would also argue (to a lesser but significant degree) that good software will try to be more tolerant of database corruption, by for example, doing an automatic rolling backup of any easily corruptible database files.

My bad for not having a UPS on a system that needed it. Vitamin D's bad for not making it clear a UPS is required, and that all files will be deleted when you restart Vitamin D after it detects corruption. That's all I'm saying.

Vitamin D is awesome. Unfortunately given the delay in response here and some of the other comments in the forums, it appears Vitamin D is in "hibernation" in terms of product development. I'm genuinely sad about that, it seems like *great* software that just needs a bit more maturity. As a fellow three-time software entrepreneur and founder (including a good old fashioned dot-com era startup, and my current company which is a Gartner leader in its field) I wish I could help.

Best,

-J


-J

[Semper Vaporo]

I have been using VDV for quite some time now (going on a year and a half??? wow where has the time gone!). Initially I was running it on a laptop with battery backup, but since the conventional wisdom is to not leave the battery in the laptop when running on AC power, I took it out. That was in the spring when we have lots of storm induced power failures around here. I also have been guilty of knocking the power cord out of the laptop or pulling the AC plug deliberately/accidentally because I got hold of the wrong power cord. I am now running VDV on a tower PC without battery back up and have had two power glitches that shut it down, and that same stupidity on my part of pulling the wrong cord out of the power strip (as well as dropping a book on the power strip On/Off Switch)!

The worst I have seen from this is the loss of just the most recent clips for each camera. Thus I SOMETIMES have lost up to 4 or 5 clips from the end of the series at the moment of the power failure. When I restart VDV it starts adding data to the end of the databases, but when I open the review mode there will be 1 to 5 clips listed as "Video file not found" and newer clips may have already been generated after those. All I have done at that point is tell VDV to delete the "missing" clips and all is well. I can see clips from before the power failure and after it. The most I am missing is less than 15 minutes from the day for each camera, and sometimes some of the cameras are not missing any time.

On the other hand, I do experience lots of False Positives, as well as complete failures to detect gross motion. Shadows of trees waving in the wind, cars in the street and no where near my defined area will trigger a motion saved clip. And yet I can walk into the field of view and do jumping jacks smack dab in the middle of the area I have defined, or walk back and forth across a vertical line I have defined, and VDV is blissfully unaware of my presence.

I think the problem with detecting motion across a vertical line is due to the the object pausing over the line or passing it too slowly. On the camera that faces my porch, I had to move the line to a far edge away from the door itself because it would miss motion if the person paused at the door to open it. By moving the line to the outer edge of the porch it detects people crossing the line because they usually do not pause there.

All my cameras are different brands; a couple of them will periodically make a large change in the brightness adjustment... one will trigger a motion detected event on each change and the other will not. To MY eye, I cannot detect a difference in the way the abrupt changes in brightness are made; the image just pops from dim to bright or bright to dim, but VDV certainly does. Thankfully the one that does it, only does it 1 to 3 or 4 times per day. If the other one did it I would get hundreds of events recorded (right now it is shifting back and forth between two levels of brightness anywhere from 3 or 4 times per second to once or twice a minute). In about 1/2 hour, the sun will have moved enough that it will stop making the brightness adjustment and settle down.

I have not used lots of different motion detection software, but of the ones i have tried, VDV is a joy to use. It is easier to review clips and define what constitutes motion (even if it sometimes seems to ignore my definitions and creates a false positive or misses something that "I" think is so obvious).

I do wish the company would become more active in producing updates and enhancements.

[JPElectron]

A few clarifications...

I am not a VDV employee - I'm a user just like you.

A forum (such as this) does not indicate to me that it is any kind of official support channel, sure I've seen some VDV employee's post in here, but I have the expectation that this is a user forum, if you want support and a response from them, there is a contact email on this page: http://vitamindinc.com/support.html

Perhaps the other thing you were running on your Mac Mini doesn't react the same way because the database (if it has a database) is not really always open/in use - but rather only accessed when something happens that causes data to be written in/out - I'd expect that VDV is different in that the DB is always open cause a video event is quite common. I work with allot of higher-end DBs (GB and TB in size) and can tell you without a doubt, closing a database dirty results in corruption, you need to take steps to avoid that ever happening.

[ryan]

Hi visionik,

I'm sorry you've had problems with DB corruption. I don't believe it is as common as you might think however (in fact I've only ever heard of a few cases). The computer I run VDV on loses power on an almost weekly basis (my apartment has poor wiring and trips fuses like crazy) and I have never suffered a database corruption.

If you really want to back up the important VDV files I would suggest backing up the entire Vitamin D Video directory. And absolutely, as you've noted, no VDV processes can be running when you copy these back or you will almost certainly get wacky results.

Sorry for the delay in replying to your posts, as JPElectron noted (thanks!) the forum is very much intended as a community hub, email and the in-application bug reporting are our primary methods of support and receive quick response.

I hope that you haven't run into any other further corruption issues, and I hope VDV is continuing to work well for you.

Thanks,
- ryan

[visionik]

Unfortunately my database corrupted again yesterday. The system is now on UPS, but the power plug was accidentally pulled from the Mac Mini by our house cleaner.

I really like Vitamin D other than this issue. I hope you also recognize some of my comments about ways you can improve the user experience when this situation *does* happen.

I will work to create a Vitamin D backup script this week.

Best,

-J

[ryan]

Hi J,

I'm sorry this happened again! That is very odd and unfortunate - I've actively tried to make this happen on numerous occasions, ever think about being a software tester? I'll definitely take your feedback into account. If you'd like to submit a bug report from within the app I can take a look at your logs and see if anything appears to be out of the ordinary in your logs that could be contributing to the problem. If you do so, mention your name from the forum and my name so the report finds its way to me.