[BRX]

In addition, it would be nice if I could say: copy (not sync) the complete database from this device to the slave. This would be equivalent to (but much easier than) making a full backup on the master, copy it to shared storage, and restore from the shared storage on the client. When you assign master and slave a uuid you can verify that the client has a sync of the master, and therefore you can use song uuid for matching. You may even disallow synching with an unknown client.

[1] An exception may be the physical file paths but there's already a path mapping facility.

Yes, that's what I'm looking for mainly, as I mentioned before. And an automatical path mapping according to the folder settings in the client would be nice, too.
The merging in both directions and checking everything is great but I fear it's so time consuming with big databases that I personally will keep my "master-slave" approach, work on the database only on the master and get the devices synced quickle with something like the aforementioned "copy db and map paths" function.

[Zubersoft]

Okay, so you guys just want an option to replace the library on one device with the library on the other device, correct?  That will require some interesting changes to ensure that files aren't left behind on the destination device (I'll probably run the clear library logic first, then just do a full one directional sync).

Mike

[Zubersoft]

There is enough work involved in adding a new mode to do a library replacement (it's almost an entirely different kind of sync requiring a new implementation), that I don't think I can add it as part of the initial release.  Instead, I want to propose something that would accomplish the same goal, but require fewer changes. If I just add a new merge behavior to "always use server data" or "always use client data", you could control what is used in each conflict case. That effectively makes it a replace, which is what you want, and then I can just utilize the existing framework. How does that sound?

[Zubersoft]

I'm going to need some help reproducing the issue when multiple songs have the same title. I first tried restoring Johan's library to two devices, then performed a merge to see if any conflicts would be generated, and the merge completed without doing anything (as would be expected). I then tried multiple tests including:

1) Copying a song, and changing metadata in the second copy
2) Importing a file with a unique name, then renaming the song to match an existing song

In the case of #1, the client device incorrectly updated the existing song when it processed the copy. I haven't thought through how to handle this yet, because I need some way to uniquely identify songs, and title + file won't cut it in that situation.  If songs with matching titles and files are used, I may have to prompt the user to pick the correct song to merge if multiple matches are found. If people don't like that, the only other option I can think of is to match up the songs based upon their index in the list of matches, sorted by database id. This would mean that the oldest song on one tablet would be merged to the oldest song on the other tablet, and each newer song would be merged to the newest on the other, with the newest ones being added if no matches are found (i.e. if there are four copies of the song with the same titles and files on one tablet, but only two on the other, the newest two would be added to the other tablet). Thoughts?

I definitely can't get a merge to fail though, so if anyone can help me set up a test case where the merge fails (or provide me two library backups, one for each tablet, that demonstrates the issue), I would really appreciate it. 

Thanks,
Mike

[sciurius]

It seems to me that after the initial tests it becomes clear there are some aspects that need rethinking. That's fine, that's what these tests are for.
Identification of songs across devices is crucial but the current approach is unfortunately not sufficiently fullproof.
If you agree that synching between arbitrary song libraries has little or no use, assigning uuids to the library and the songs may solve most, if not all, problems.

I'm a bit surprised to hear that a full replace using synch would require a lot of additional work, since that logic is already part of the library backup/restore functionality.

[Zubersoft]

It's only not a lot of additional work if I don't try to resolve file deltas between the two. If I just copy the database over from one device, and then copy every file over without worrying about whether it actually needs to be copied over, and I update the file paths using similar logic to a library restore (although I'd have to support the path mappings), then the logic doesn't get complicated. If I instead try to do comparisons of every file on one tablet with files on the other tablet (accounting for differences in paths) in order to avoid doing unnecessary copying, then things get more complicated. The same is true if I have to figure out which files are no longer needed after the sync completes to ensure orphan files aren't left behind. One option would be to perform a clean library first, but that can take awhile on Windows 10 (it's not too bad on Android). So the question is, if I try to support that kind of library replacement, how smart do you want it to be about handling files on each device? That will dictate how much work is involved and when I can support it.

As far as identifying identical songs on multiple devices, assigning uuids isn't really sufficient either. Two users with separate libraries may import the same file off Dropbox let's say, then add their own annotations and other fields. If they merge their libraries, my logic should handle the fact that they both created the same song (assuming they used the same title), and merge it. If a UUID was assigned, each device would have assigned a different UUID, so the two songs wouldn't be considered the same. Using something like a UUID only works in the case where you have devices with identical databases, in which case I can just use the ID column of the songs table as that's unique and would be identical. I'm actually already using the song ID if multiple matching songs are found with the same title and files and the device databases happen to match, so I think I've already handled that case. I don't think I can make the assumption that users who use this feature can be expected to have synchronized the libraries on the devices at some point beforehand using a library backup/restore.  This is why I think, in the case that libraries are being merged with very different databases, that it's acceptable to prompt the user to pick the correct song (and perhaps use index as the default selection). This will encourage people who want to use this feature to not have songs with identical titles and files in their library, or for them to use a library backup/restore first to avoid conflicts. 

One possible way to handle this issue would be to add a new song number field that has to be unique. Users could then assign the same numbers to songs in different libraries, and the merge could use that as the basis. This would need to be a new option in the merge, perhaps called "Match songs using number". This would require a lot of work for users to have to go through and assign numbers to all their songs though, so I'm not sure that's a great answer either. 

Thanks for the feedback,
Mike

[BRX]

Hm, I don't have any ideas yet.

But Mike, maybe you can elaborate on the process MSP does right now with your library sync (copying, comparing, merging, which data/files,tagging). 
Maybe it's clear for Scirius, but I'm actually not quite clear about what MSP does when I start the sync.

[sciurius]

Two users with separate libraries may import the same file off Dropbox let's say, then add their own annotations and other fields. If they merge their libraries, my logic should handle the fact that they both created the same song (assuming they used the same title), and merge it.

I beg to differ. Even though they both used the same file to start with, each user created a unique song. I personally doubt it has any use to merge. For example, if it is a chordpro file, annotations are relative to the rendering of song lines, and that depends on individual settings like text size and such.
If one user replaces the file by a totally different one, would you still consider it to be the same song?

One possible way to handle this issue would be to add a new song number field that has to be unique.

UUID?
If you make it an editable property the user has ultimate control over whether two songs are to be taken ‘the same’, or not.

[Zubersoft]

I can think of plenty of cases where everyone in a band imports the files for the week off Dropbox and sets up their libraries. Maybe one user is in charge of setting things up for the week and then syncs to the other tablets. If one of the other users jumps the gun and imports the file to practice it beforehand, the library sync shouldn't treat that as a completely different song from the one the band leader created. If you disagree with that, then you've basically said that the library sync feature only makes sense for users with identical databases, otherwise it can't be used.  As far as replacing the file with a different one, the current merge logic tries to find matching files, but if it can't, it makes due with matching titles and considers the songs the same.

Whether the user enters a custom number or a UUID (which would be pretty clunky as those are typically 16 characters long), the end result is the same. This puts a lot of burden on users with large libraries, and would be tedious for users to manage. I'm not sure how often this really matters - are we trying to solve a problem that is going to be extremely likely with most users libraries, or is it more of a fringe case? While I can see libraries having multiple songs with the same names, I would expect most people have different files for those songs. If users are making copies of existing songs just to have different annotations, notes, etc, then that is the one situation where I won't have a good answer except prompting for a decision (and hopefully that will encourage users to uniquely name their songs to avoid further prompts/conflicts).

MSPs sync in a nutshell goes like this:

1) Server sends database, file path mappings and its storage directory path to client to start the merge. 
2) Client sends same information and database to server and a list of all file sizes (so that server doesn't have to request this later when looking for differences)
3) Server iterates over every song in its library to see if a merge is necessary. To find a matching song, from the other library, title is compared first, then file second, and anything with a matching database id is preferred.
4) The merging of a song does the following:
    a) Checks files for differences and if a difference is detected, either enqueues the file to be sent from the server at the end of the merge, or requests the file from the client right away
    b) Checks file properties for differences (page order, text display settings, etc). Files must always be merged first because other things like the properties per page depends upon them being the same for each song
    c) Metadata is checked next, which covers most of the data in the song (group assignments, links, bookmarks, scroll settings, duration, rating, cropping, rotation, etc)
    d) After metadata, MIDI commands are synchronized
    e) Next comes annotations
5) If a song doesn't exist on one of the devices the song is either added or deleted depending upon one-way vs two-way sync
6) Server now iterates over all client library songs, processing songs that weren't already encountered. These songs are added or removed depending upon the merge direction
7) Server notifies client that it's waiting for it to finish sending every requested file (the next step requires the client to be finished and ready)
8) When client is ready, the server writes all of the changes it has made to the databases. This persists all of the changes encountered during merging in both the local and client databases. 
9) Next, groups are merged (it's important this happens at this step so that all metadata has been processed for individual songs). This typically just results in setlists being merged, as the other metadata should be the same
10) The updated database is sent from the server to the client, which closes its database and opens the new one
11) The server sends over the list of songs for the client to delete (if any were marked for deletion during the merge)
12) The server sends over all of the enqueued song/audio files. This happens late in the merge so that the client has all of the updated database fields from the server indicating where files should be placed and the correct songs to associate them with.
13) The server sends a notification to the client that it is done
14) When the client is done receiving files, it sends a notification to the server that the merge is complete

Is that basically what you wanted BRX? I should mention that the merge process checks literally every field the application supports when looking for differences, so there is a ton of code, but it all runs quickly as it's light-weight.

Mike

[BRX]

Yes Mike, that's what I meant and it is very helpful for me to understand the process and to see where my syncs get stuck.

I reckon you didn't find the problem or a solution for the sheets directories on external SD-cards > Android 6.0 ?

[Zubersoft]

I'm still working on it - I was trying to address issues with songs with identical titles and/or files first.  I'm still making a few changes for that, then I'm going to focus on figuring out the SD card issue. 

Mike

[sciurius]

Thanks for the extensive info on the synch process. Much is clearer now.

[BRX]

If you click the path field, does it not pop up the Microsoft file chooser so you can pick the folder? That's how it is supposed to work. 

I think you are seeing that issue because you are using a removable SD card for your storage location on Android 5.0+. I only have one device with an SD card that I can test with, but I'll see if I can reproduce what you are seeing using that.

I'm back at my W10 desktop. Again, the Microsoft file chooser doesn't pop up. I can edit the name field but not the folder path.

Did you mean this problemis Android 5.0+/SD Card related? Very unlikely since this problem is there without any connection to an Android server and W10/MSP started as standalone. Also there are no SD cards in my desktop. (I will test on my Surface which has a SD card I want to have the sheets folder on, but didn't get to that yet).

[Zubersoft]

BRX - I will fix the issue with that dialog on Windows 10. No, it's not related to using an SD card. The other issues you saw on Android are related to using an SD card.

Mike

[Zubersoft]

I've released new versions of the beta for Android and Windows 10 (v2.2.1). Changes include:

1) Better handling of songs with duplicate titles/files. You will be prompted if MobileSheetsPro can't decide which is the right one to use.  Future choices will ignore previous choices as possibilities, so if you only have two songs with duplicate titles, you will be prompted only once for a decision.
2) Fixed bugs that could cause file merging to occur when there were no actual differences.
3) Added progress bar for tracking overall progress. Decided against adding two separate bars (current vs overall) as it greatly increased complexity and provided little usefulness in most situations.
4) Fixed issues on Windows 10 when paths from Android are used that are invalid. All paths are corrected for Windows 10 if needed now
5) Fixed issues where the same group could be added to a song multiple times which shouldn't be allowed
6) Added new merge behaviors (only use server, only use client)
7) Added option to merge dialog to use selection for all future merges
8) Added prev page/next page labels to cropping editor arrows so it's clearer that pages can be turned
9) On Android, highlight annotations are now placed behind content on the PDF. Added option to enable/disable this while exporting annotations to file as this isn't supported with normal PDF annotations, so the original PDF has to be rebuilt with new images per page.

If beta testers can test both the new library sync features and the highlight annotations, that would be great.

Thanks,
Mike