[Zubersoft]

As a side note, I've heard multiple reports now of database problems due to the companion app, and I believe it may be due to the way I'm copying the database file from the PC to the tablet. I think what I'm currently doing is inherently unsafe. SQLite uses some advanced techniques to prevent database corruption, including the use of a journal file, which can delay when data that has been written to the database is actually present in the database file (transactions all go through the journal file first). I think it's possible for the companion to be sending the bytes of the database file when the file isn't in a proper state for that to be done. I'm going to try changing the companion app to use the SQLite backup api to create a copy of the database that is safe to send to the tablet. I will probably also change the tablet to clear out any journal file that is present if I replace the database, as I think having an invalid journal file present could also cause problems. I will also be working on adding more protection against possible disconnects while file transfers are still active.

Mike

[GraemeJ]

Now I'm over the panic stage, it's good to know there was nothing wrong with the backup file itself (that could have been a disaster, although I always keep the previous three or four, just in case) and that recovering the situation was relatively easy and requiring no more than a single tablet.

Interesting that Mike is questioning the way the new Companion is structured. I never experienced any problems at all with the old one and I've had MSPro running on two (sometimes, three) tablets ever since its first beta release - yes, it had its problems, as one would expect for a beta product, but never something like this - and I suspect the Companion is the real culprit.

Thanks for all the support guys and I think the motto has to be 'proceed with caution'.

[sciurius]

As a side note, I've heard multiple reports now of database problems due to the companion app, and I believe it may be due to the way I'm copying the database file from the PC to the tablet. I think what I'm currently doing is inherently unsafe.

I'd say disconnecting (closing) the database before copying should be sufficient to make sure it is in a consistent state.

[sciurius]

I'm happy to share the internals of the .msb file structure with you if you want to develop that tool further. It's not too complicated. I'll send you a PM with the details if you would like.

Great, thanks!

... and it seems like this would make it pretty easy to create a backup file in a zip file format. I think I will add this as an enhancement - the choice to back up to a zip file or msb file. I could also then offer the ability to compress for those that want it.

Nice idea, but why maintain two formats? Create the newer backups in zip format and accept the old msb format for restore only.

Also, offering the ability to compress seems unneccessary to me. In general, you can compress text (ChoPro) and midi files. All other formats (PDF, MP3, ...) are known to compress badly.

[Zubersoft]

The big issue I see right now with using a zip file is when it comes to incremental backups, I will have no way to edit the existing zip file to append new data. The Android ZipFile class does not support this, and I haven't found an open source library for Android that will let me modify an existing zip file. This would make my idea for incremental backups a lot more messy, because I would need to implement a model where I chain together incremental backup files, instead of just being able to write the incremental backup to the end of the existing backup file. If you have other ideas on this issue, or suggestions for an open source library to look at, let me know.

[sciurius]

Personally, I don't think that incremental backups add much value. I prefer complete, consistent backup sets and storage is cheap.

Having said that, appending data to an existing zip is a bad idea. It's always better to create a new zip, copy the old contents, and add the new data.

[GraemeJ]

Personally, I don't think that incremental backups add much value. I prefer complete, consistent backup sets and storage is cheap

I was the person who originally raised the issue of incremental backups (and I thought it had been abandoned until I saw this post). In some ways I agree with you and complete backups are probably better (and memory is certainly cheap).

However, on the down side, it does take a loooong time to make a backup of any size in MS/MSP. My own backup is 1.5+ Gb and it can take 40 odd minutes to do a full backup/restore.

[sciurius]

My own backup is 1.5+ Gb and it can take 40 odd minutes to do a full backup/restore.

So it's just a matter of time that you hit some stupid limit (2Gb?) and everything starts blowing up... Having a single-file backup of that size doesn't feel reassuring to me...
Not to mention the time it takes to copy files of this size over the network to background storage.

May I suggest a slightly different approach, based on my experiences until now.

I have several collections, all for different occasions. For example, I have a collection with repertoire for my folk band, one for the pop choir, one for the aphasia choir, and so on.
I find that, when on one occasion I hardly ever need the other repertoire, if at all. So for me it would nice if it were possible to select a collection at startup and then no more worry about the other songs. (Also, some collections have the same songs in different arrangements, so this would lessen the confusion as well.)

As for the backups, MSPro could be made to backup a single collection, making them smaller and easier maintainable.

Alternatively, like most tools that make large backups, store the information in a folder instead of single file.

Also alternatively, leave backups to tools that are specialized in making backups.

[sciurius]

BTW, I was a bit surprised to find out that MSPro includes all song files in the backup set, even though I asked it not to manage my songs.

I have mixed feelings about this... On one hand, it is good to have a consistent backup set that includes all, on the other hand, when I say I'll manage the song data myself, I'll manage the song data myself -- including backup and restore.

[itsme]

I agree with Sciurius that a separate backup of the song files and the MSP database would make sense. I already thought about just copying the database file ('Expose Database File' option) manually. There's a full mirror of my song files on the PC anyway.
Some thoughts around such a scenario:
1.) Backing up the settings would be convenient. Why not storing them in a special table within the database?
2.) I already posted a restore error and proposed a more detailed error message
see http://zubersoft.com/mobilesheets/forum/...p?tid=2396
The error itself was probably caused by using a folder on the external SD card different to the specific MSP directory 
<extSdCard Path>/Android/data/com.zubersoft.mobilesheetspro/files
as explained in http://zubersoft.com/mobilesheets/forum/...p?tid=2522
I will consider using this path, but anyway...
3.) And I made a proposal for a function to check if all files are available
see http://zubersoft.com/mobilesheets/forum/...p?tid=1950
Implementing that would be VERY convenient not only in various backup and restore scenarios but also in many other cases.

[Zubersoft]

I used to provide an option just to export the database file, and you wouldn't believe the number of emails I got from people who backed up their library using just a database backup (thinking they were backing up everything), even though I put a huge number of warnings in the description. I decided that it was too confusing for most users to provide both options, and now just have the "Expose Database" option for those that want to have direct access. If all you want is a database backup, you can just copy the .db file to a place of your choosing.

Thinking more about the .zip versus .msb issue, there is one issue I don't have an answer for yet. People like to restore their backups from the companion app, and when I do this, I stream the data across so that the tablet can extract the backup without needing adequate space to copy the file across. From what I've seen so far, I don't believe the Java zip classes will work with a stream of data - they all seem to want the file available locally. Another issue is speed - from what I've read so far, the Java zip libraries sound like they may be pretty slow. I'm going to have to do some measurements, but I wouldn't be surprised if my msb implementation is much faster to backup and restore. This only matters for large backup files of course, but I'm thinking it still makes sense to provide the option to choose one or the other.

itsme - I haven't forgotten about those requests, I just haven't had time to address them yet. As far as moving all of the settings to the database, I may consider this, but it would mean that every class in my code that wanted to persist settings would have to have access to the database. Using preference files is just much simpler. I have already researched how to back up all the data in these preference files, so I should be able to place these in the backup without any issues.

Mike

[sciurius]

Thinking more about the .zip versus .msb issue, there is one issue I don't have an answer for yet.

Using a zip instead of a custom format was just a question/suggestion. If it opens a can of worms just forget it. There are more important issues to address.