[semmelbroesel]

Hi.

I am trying to add a folder of about 10,000 tunes to MSP to my Android tablet using the Companion and Batch Import.

Once I am done, none of the imported files work - the loading symbol keeps rotating.
I can click EDIT and get meta data, but as soon as I check the file, it again just spins.

I made two attempts - first, I took the entire folder and ran it through Companion Batch Import, and I ran into the issue I mentioned.
So I deleted all 10,000 tunes again (I had imported them with a specific Collection, so I selected all Collection songs and deleted).

I just finished doing this again, but one letter at a time (i.e. all tunes starting with A, then B, etc.). So I still have the 10,000 tunes in my library.

And now I cannot open any of them.

Worse, my original files no longer open, either.

I have closed and force closed MSP twice.

Some files have special characters in the file name, e.g. an apostrophe, and a few even start with an apostrophe.

While I only had the files starting with A (and "Apostrophe-A") in there, they opened fine.


Another thing I did that might have broken something:
I selected all 10,000 tunes and hit SHARE to create a file for Google Drive.

Each time I have done this, it stops at some point and no longer moves, and I cancel, and after that I cannot access the files as described.

I don't know which part causes the issue.

Any ideas?

And any ideas on how to get these files into my library and into an exported MSF file for Google Drive?

I guess in the meantime I'll try deleting them all and importing them again to see if I can reproduce exactly when they stop working.

Thanks!

[semmelbroesel]

Update:
The Companion apparently can read the files when I connect it to the tablet, but the tablet can't.

I have tried the fixes for missing files and broken paths, no luck.

Also, when I told Companion to delete the tunes, it started rattling through my hard drive and 10 minutes later had to be forced to shut down.

I'll try deleting the files directly on the tablet and start over and see what happens.

[semmelbroesel]

After deleting the 10,000 tunes, I still cannot access any of my files on the tablet :-(

I thought maybe if I uploaded one tune through the Companion, the database would get synced and fixed or something, but no dice.

So right now I need urgent help getting my files to load again - I need the tablet for rehearsal later tonight...

[Zubersoft]

So the first thing I would like you to try doing is rebooting your device. If there is an I/O issue (such as folders/files being locked), this can remedy that. After doing that, please try loading a song that uses a PDF and verify it works correctly. If it does not, please go to Settings->Display Settings->Render Preference and try switching between quality and speed to see if this makes a difference with the song loading. If your PDFs are loading correctly, the next thing I would like you to try is copying over a PDF to the tablet itself and using Import->Local File or Import->Batch Import to verify that the file imported off the tablet works without issue. If that works, then connect to the companion app and try the batch import again. If you import all 10000, and then the tablet can't load again, try rebooting the tablet after this to see if that fixes the issue. That is a pretty large number of files so I'm wondering if you are running into issues with MobileSheetsPro running out of resources (memory, file handles, etc) if it's parsing those files for information. While I have batch imported thousands of files, I can't say I've ever imported 10000+ at once. 

I have also never tried using Share->Export as .msf with 10000+ songs at once, because that functionality wasn't really intended to export basically an entire library worth of songs at once (although it should be capable of it, I've just never tested that). The same problem may be coming up with this as with the batch import from the companion. I'll have to see if I can gather together 10000 files to test with. I think at the moment I have 3000 of my own, so I'm not sure how easy it will be to gather up 7000 more.

If the batch import from the companion is not going to work, even after performing smaller batches, then the next step would be either to import directly from the cloud or copy the files to the tablet itself and import from the local storage. 

Mike

[semmelbroesel]

Thanks for the quick response!

First off, you were right about rebooting, I can access my regular files again!

So now I'll try importing the whole bunch again, and if I have trouble again, I'll reboot, and hopefully, the files will still work.

And the reason I am trying to export this many files is because I am in two bands who (on my recommendation) use MSP, and we just received new material, i.e. 10,000 tunes, so I'd like to share the tunes with others in the band. I might have to split it up, but first let me get them back on there to see what happens :-)

Thanks again! I really appreciate your work!

[semmelbroesel]

Thanks to knowing that rebooting the tablet fixes the issue of not being able to open any files once an issue occurs, I've been able to carry on.

Next issue:
I am trying to upload the 10k tunes to Google Drive, and it fails way before the upload.

It gathers the files together into one large MSF file, and that's where it has trouble.

The process simply stops at 4%. I went through all the files starting with the letter A to test if they worked. After that, sharing to GDrive got to 20%, but it always stops there. The software still reacts, but once it reaches this point and gets interrupted by me canceling the process, it no longer can read any of the files.

I am wondering if one of the PDF files has an issue, but with 10k of them, I'll be sitting here until Christmas trying to find the broken one.

I tried missing files and broken paths, and they found nothing.

Is there a way I missed to run through all files and test if they can be opened?

Or is there another way I can get this to work? I don't think it has to do with GDrive because it never reaches that stage, and I can't really know for sure which file is at 20%.

Thoughts?

Thanks!

[semmelbroesel]

OK, so here's a weird observation:

I tried splitting the tunes into smaller groups, so I tried sending 2,500 tunes into GDrive. The file creation stopped at 96%, multiple times in a row (with a reboot in between each time because it locks the entire library after each failure).
So I went through about the last 150 tunes on the tablet, i.e. scrolling through them all.
I found no issues.

Then I tried uploading again, and this time it worked.

So now I'm wondering if some files only get fully "activated" once you view them or load them at least once.

I'll see if I can find something like a "slideshow" feature that will briefly go through all tunes because clicking about 20,000 times (some tunes have multiple pages) is not going to be how I will spend my weekend :-)

Still hoping for better ideas - just wanted to give more data.

[semmelbroesel]

For now, auto-scrolling on 1 second might do the trick to get all tunes pre-loaded once.

[Zubersoft]

I don't think you should have to load them for this to work. I want to help test this, but even with 6700 files, I'm already over 4 GB of data. At that rate, I'd have a library over 6GB at 10000 files. Uploading 6 GB of data to Google Drive would take a very long time. How much data is used by the 10000 files in your library?

Thanks,
Mike

[semmelbroesel]

I think it's about 6GB of data, well calculated :-)

But again, the issue is not in uploading - the file never gets created.

The creation of the MSF file in the cache folder is halted at some random spot.

It has to be something weird about the files I am importing, but at the same time they are going through the Companion with automatic cropping.

All I know is that the files starting with A-D did not finish creating as a file until I ran them all through the auto-scrolling.
So far, I have A-D in one file and one next batch (don't remember the letters).

So to clarify my steps:

I went through the Companion to separate all the new files into new Collections, e.g. all tunes starting with the letters A-D were in one Collection, and so on.
I tried uploading the first collection at least 4 times, and each time it stopped creating the MSF file, always in the same spot. I had to reboot the tablet in order to load any files in MSP after it stops.
Then I ran through all the files in auto-scrolling.
The next time I uploaded the first collection, it went through without a problem, so did the second batch.

I think maybe there is something funky about my files that MSP has trouble with until it actually opens and displays them - until then, there are funky files it cannot export.

I can't really explain it, but this is what I have gone through, and I have had the same issue with other files.

It could be the source of the PDF files - they come from the same person who scanned them.

[Zubersoft]

Android only supports 4 GB files (a limitation of the file system). MobileSheetsPro will be unable to continue writing to the file and will encounter an error. I don't have a mechanism for splitting msf files across a 4 GB boundary (same with library backup files). 

So I will run some tests with 2500 songs as that is more in line with what you are attempting at the moment. 

Thanks,
Mike

[semmelbroesel]

Oh, I did not know about the file size limit... It seems to work fine otherwise.

Maybe I need to import my entire library again but convert from PDF to image - would that reduce the file size?

I could probably do without the new 10,000 tunes, but it would be nice to have them, all the same, as one of my bands refers to the occasional file in it.

[Zubersoft]

Most likely not - PDFs usually use decent compression algorithms, especially for grayscale or black and white images. One thing you could do is sync your library to the cloud using Sync Library->Sync to a cloud folder, and then your band mates can just sync to that folder and it will pull in all 10000 songs. If you need to limit this to just the new 10000 songs, use Settings->Library Settings->Switch Library to set up a temporary library first that you can clear later. Alternatively, you can use Sync Library->Sync to another device to just sync your device to theirs over WiFi, which would be faster.

Mike

[semmelbroesel]

I haven't had luck with Sync - it is really slow (even on fast internet) and typically stops at some point.

By the way, I had a misunderstanding in my previous message: I thought the 4GB limit meant the library size, but I think you actually meant the size of the export file.

So far, I have had luck creating about 1GB files by splitting the collection into smaller collections based on the starting letter, and I have 60% of the new files uploaded by now.

[Zubersoft]

Yes, I meant the export file size. I'm glad you are having success with smaller collections, but I'm sorry it's such a tedious process. The sync is slow mainly because uploading to those cloud providers through their interfaces is slow. Using a device to device sync would be much faster.

Mike