[Taxel]

I was just in the process of uploading my library to google drive when for some reason the sync stopped.
When I restarted the sync, the correct error message showed up: "Unable to find library in the selected folder. Would you like to upload your device's library for future merges?", which I confirmed.
It then just prints out the error "Received database version 00 but expected 55. MobileSheetsPro must be the same version on each device. Ending sync." and does not start the sync.
I'll just delete the folder and start again for now, but since I can expect the upload of my library to take at least the next 10 hours if the sync fails again and I get this error again this will be very frustrating. 
Is there maybe a way to export the library with the exact file structure and metadata to my PC and then upload it from there, where I can just resume the upload? Also, could I just sync some of my library instead of all of it?

Thanks in advance!
Taxel

[Zubersoft]

How large is your library? How much free space do you have in your Google Drive account?

I'm not sure why you are seeing that error. That usually only occurs if an invalid database file is present in the sync folder. If you clear mobilesheets.db and the hashcodes file out of the sync folder, in theory it should work.

If you want to use the sync feature, you'll have to rely on MobileSheets populating the folder, setting up the database file there and creating the hashcodes file. If you want to switch to just using a file sync approach as others have done, you can do that instead. You'll just want to manage your own files and folders. On the Windows 10 version, that means using a folder of your own choosing for Settings->Storage->Set MobileSheets Storage Location, although you'll have to synchronize mobilesheets.db to the default storage location, as it can't be moved anywhere else due to the C SQLite library not working well with UWP due to file permissions. In the Android version, disable Settings->Storage->Set MobileSheets Storage Location and just manage your own files and folders.

Mike

[Taxel]

Hey Mike,

My library contains 4200 songs, the folder on the SD card is ~9GB.
My Google drive shows 3.3GB of 15 GB used so there should be more than enough space left.

The upload fails on random files after a couple of hours with a nondescript error, so I guess it happens because of a network error. I can report some more about the "Received database version 00..." error:
It appears that this error occurs when trying to restart the sync after it failed. An easy way to fix it is to close the app and reopen it, this lets me start the sync one more time (but starts with the first file again, so it never completes).
Unfortunately, this issue is a real blocker to my build-server project so if you could add a checkbox "skip existing files when syncing" or something like that in the next update you would be my hero.

Using the built-in sync functionality would be preferable to me because otherwise, I have to find a way to manually download the files from Google Drive which would probably need more user interaction than just selecting "sync with google drive" and be done with it. Is there maybe any documentation on the hashcodes file? I am seriously considering creating a script that uploads the files to Drive, creates the hashcodes and changes the paths in mobilesheets.db

[Zubersoft]

It's not that simple I'm afraid - I can't know if the files that have already been uploaded to Google Drive were part of a previous sync or were just manually copied there. Also, while I can check to see if a file name exists at a given location (meaning MobileSheetsPro may have copied the file there previously), I can't know if that version of the file actually matches what is on the tablet without doing a hash calculation, which requires scanning through every byte of the file (which would take a considerable amount of time). If I just ignore the potential for the files being different, then sure, I could skip the files. The problem is, what if you create new versions of files on your device or edit existing versions and want those files synchronized to the cloud? While I can (and do) perform simple checks first such as seeing if the file size doesn't match, I can't rely on timestamps because the cloud timestamp won't match what is on the tablet (the hashcodes file was created for this reason). If the size matches, I have to do the hash comparison. 

One thing I can do is to try implement a framework for resuming the upload, where I update local files that indicate how far along I am with the sync, along with a copy of the database that will be uploaded. If the sync fails and the user is given the option to try and resume, I could then read those local files, figure out how far along the upload was, and just pick up from there. I'm not sure exactly how much work this would be though. It certainly would require changing a bunch of code to allow a sync to be resumed, which there is no concept of right now. I really need to get the annotations rework done though, so I can't get sidetracked with any other efforts right now.

One thing I can change is I don't currently retry uploading files through Google Drive if the upload attempt fails (as it's supposed to be reliable in nature). I can change this code to attempt 3-5 times, and then if it fails, the sync would fail. This could be enough to get the upload to finish successfully for you. I have personally uploaded 5GB libraries to Google Drive without any issues, so it's certainly possible with the current implementation, but it requires a connection that is never interrupted. If you would like to try a test build that retries the upload multiple times before failing, send me an email at mike@zubersoft.com.

The hashcodes file has 4 entries for every file in the ilbrary:

String that is the path to the file
32-bit integer hash calculated using murmur3_32 from Guava's Hashing class
64-bit long that is the last modified timestamp (should match the last modified timestamp of the file on the Android device)
64-bit long that is the file size

You'll have to make sure that the paths are setup correctly in the database in the cloud. They have to match what MobileSheetsPro is going to expect. I would suggest synchronizing a small library to see exactly how it's populated before trying to write a script to populate it yourself.

Mike

[Taxel]

Unfortunately even with your second test build and all power saving mechanisms turned off, the upload did not work on my Fire HD 10 tablet. The second build never spits out an error message and when I try to cancel the upload it shows "Merge is still in progress [...]" but it simply does not upload any files (which I only find out when I look at the timestamps)

I will try it with my samsung phone next, but if that fails too, I will have to learn how to add files to Drive that are recognized by MSP myself. Luckily that's something I want to do anyways so I'll keep my small testing library on my phone for the time.

I seem to be on a promising path; I can already load the Database via a button, display all songs with artists etc., rename a song (which wasn't as easy as it sounds since for even this change just renaming the DB entry does not work) and delete a song.

I have encountered another roadblock, though: I am not sure how I can recreate the hashcodes.txt - reading out the filepaths, last modified timestamp and fileSize is easy enough but when I try to hash the file contents myself, I get a totally different number from what is in that file.
Not only that but I strongly suggest there is something weird going on on your side since apparently murmurhash3_32 returns an unsigned 32 bit integer but three of the four hashcodes in the hashcodes.txt of my very small testing library are actually negative!

I highly doubt that I might be using a broken library, I've tried two different ones (not based on the same C/C++ code) and they both yielded the same result for my file.

Here's the example entry from the hashCodes file, generated by MobileSheetsPro:
1ozphXmH8RroGNJpbj2nGMNaJNg9vxI5q/Are You Lonesome Tonight_.jpg
-1274144557
1583326983442
177007

My calculated hash: 4275668817

[Zubersoft]

That's disappointing to hear that the upload never finished. As I've mentioned before, I've uploaded libraries over 5GB in size, but I have a gigabyte fiber connection, so that may have something to do with it.

Check this out: https://github.com/google/guava/tree/mas...ommon/hash

Those are the classes I've utilized. I call Hashing.goodFast32Hash(HASH_KEY)) where HASH_KEY is equal to 0xC58F1A7B. 

Mike

[Taxel]

I don't see  Hashing.goodFast32Hash(hash key) anywhere, just Hashing.goodFastHash(32), which returns the murmur3_32 implementation. Is the hash key the seed you're using for that?
To me it still looks like the hash function can only produce positive numbers - could I ask you to take a look at your implementation and find out how the negative numbers come to be? This really confuses me and hinders me from adding files to my Google Drive folder programmatically... Would it help if I sent you the file mentioned above via email, where we know the murmur hash (according to my library) and the one MSP produces?

Taxel

[Geoff Bacon]

I don't know anything about what you are doing/trying to achieve here but

Hash results are typically expressed in hexadecimal notation i.e + and - signs don't enter into it (a -ve sign just indicates that the output has been formatted as a decimal number)

Geoff

[Taxel]

So, Geoff, you agree that a negative hash does not make much sense?
I am not here to criticize anything about Mike's  software design, just to find out how I can transform the hashcodes I am calculating into something MSP will actually understand.

[Zubersoft]

I call Files.hash(file, Hashing.goodFast32Hash(HASH_KEY));

Using the return value from that, I call "asInt()" on the HashCode object that is returned. So it's a signed integer value (negative values are just fine). And yes, HASH_KEY is the seed value passed to the function.

As a side note, 4 bytes of data are 4 bytes of data regardless of it being a signed or unsigned value. If you take your unsigned value and cast it to a signed value to get the signed representation, if the value is greater than the max positive value of a signed integer, it will be negative.

Mike

[Geoff Bacon]

I think Hash codes are primarily displayed as hex numbers because it means they have a consistent length (8 chars). This usually makes it easier to extract data embedded in a string

Geoff

[Taxel]

I got it!
The asInt() function in the Java library returns a signed int32 in little-endian byte order. So in the node.js package I am using, the correct code to reproduce the hash is

Code:

const h = murmurHash(buf, "buffer", 0xc58f1a7b);
hash = h.readInt32BE();

[Taxel]

Another question concerning the sync between MSP and GDrive:

I am almost done! I have already managed to add files to the GDrive folder and database so that they are read in by MSP when I sync with cloud.

Only one thing: When I update the file in GDrive, update the hashfile, update the modified date in the Files table and for the song, MSP does not download the new file. I also tried creating a new song (as it is apparently needed when for example renaming a song - the songId apparently has to change) and updating all tables with the new song id but to no avail.
Could you please tell me the conditions which have to be met so MSP recognizes a new file in a cloud folder and downloads it? I probably just missed one step.

[Zubersoft]

Renaming a song does not require a new Song ID - you'd just need to update the title in the Songs table.

The database on the cloud has to have an entry in the Files tab, and that entry needs to link to a song through a valid song ID, and you also have to have entries under ZoomPerPage and Crop for each page in that file. If all of those conditions are met, then the song should be initialized in the cloud folder database and the device database will see the difference and download the file.

Mike