[Zubersoft]

I'm a little surprised by your results. I added code for the batch import so that it wouldn't even perform the duplicate file check if there wasn't a song in the library already using that file. That means if you are importing new files through the batch import, one of two things should happen:

1) If a file already resides at the output path for the file (depends upon your storage settings and storage directory), then you should get prompted to handle the conflict. You absolutely do not want to have "Update songs if matching files are found" checked. If you do that, it's going to try to update a non-existent song. This is a bug in the companion app - I just added a fix so that it won't try to do this anymore, but in the meantime, don't check that option. Instead, make sure you handle the conflict by overwriting the file, and check "Apply to All".
2) If a file does not reside at the output path, the file should be imported without any issues.

You definitely want to leave "Avoid Duplicate Songs" checked, or it will just add tons of duplicate songs in your library, which you don't want. If you don't have "Update songs if matching files are found", then I'm not sure why the batch import wouldn't import all of the files like you are expecting. Can you take a screenshot of the settings in the batch import dialog? I just want to check them. Also, what is your storage location on the tablet?

Mike

[musichawk]

MSP is set to Let MobileSheets Manage My Files, which are stored here, same location since MSP installed some time ago:
/storage/extSdCard/Android/data/com.zubersoft.mobilesheetspro/files

I use flat storage, no subdirectories for songs.

The problem, possibly existing for months or longer, is that some songs I add to tablet library via MSPC do not actually get added. I've had "I thought I had that song" moments but figured I was forgetting. I can add individual songs, no problem. But now, I think there's a Batch Import problem, because that is how I've added the vast majority of songs. I have a mix of missing songs and missing files, and can't seem to correct this via Batch Import or any other MSP action.

Using the latest updates of MSP and MSPC, with Batch Import always set to Avoid Duplicate Songs, the problem is that the two forms of mismatch do not get corrected. When a duplicate file is found, selecting Use Original File, or Overwrite, or Skip File doesn't seem to matter, at least not in my test records/files. (If I'm incorrect about this, please advise what tests to run.)

I find instances of two types of database-file mismatches, and they persist, can't find a way to have Batch Import "fix" them:

Source of files is always the same, a single folder on PC that has all the PDFs I want on the tablet. Every file is precisely named, and MSPC is set to use the Filename as the Song name. For Batch Import I simply select this folder, and MSPC seems to process all the files in the folder.

When...
-Song already in database
-File not on tablet
Result=The missing file does not get added to the bad record.
Instead, the missing file gets added as a new record, so the song name exists twice in the database. In one test, the prior file-less database record was then found to link to the newly-added PDF file. Presumably this was because the db record had originally pointed at the file, which disappeared somehow but the pointer was still there.
Note that the song name that gets duplicated is identical to what was already in the library, because it is the Filename. There seems to be no checking for an existing db record of the same name.
The problem is that duplicate records appear and then must be manually spotted and deleted. The risk is if the two (or more) records do not all point at the file, probably they do, but the only verification is to open each song.

When...
-Song not in database
-File already on tablet
Result=Nothing happens with Batch Import, the song does not get added to the database, unless....
The orphan file is identified during import via a Duplicate filename dialog. If I select Overwrite, the song gets added to the database and presumably the file gets replaced. However, with more than 4000 files the only practical method is selecting Duplicate file Overwrite to do this with all files by checking the box, which results in almost the entire library being re-copied from PC to tablet, just to get any "lost" songs added back to the database. This would take a while, but is it safe? Any other side effects?
The orphan PDF file does not show up via Find Missing Files, not sure what it actually does, though in a previous test it found orphan mp3 files.

I'm trying another Batch Import, specifying Overwrite for all files. Hoping this does not result in the topic that started this post, double records for almost every song, yet some songs skipped. Here goes...

NOPE, FAILED. Importing with Overwrite of all existing files, the library on tablet grew by ONE file. The Library still is missing more than 500 files in the PC's source folder total of 4658 files. The tablet folder that stores MSP songs has about 4940 files, apparently are on the tablet but not in the database therefore unidentified by MSP.

This is a mystery. I have never deleted a song without also deleting its file (except for a few recent tests), so how did these files get on the tablet and/or their database records get deleted? Does MSP copy new files from PC to tablet, THEN update the database, which perhaps failed for some reason? It will be difficult to delete them because they look just like every other MSP library file. This is where the solution could be for MSP having a mode where it scans all files for matchng database records.

The Proper outcome would be for ALL files from the source get loaded onto the tablet, no more or less. I see no cause for these large, persistent anomalies. Or am I thinking about this wrong?

[Zubersoft]

I'll run some more tests to see if I can reproduce what you are seeing now that I know your setup. MobileSheetsPro does send the files first, then sends updates to the database. The all happens after the batch import steps through every file though. If you caused the connection to fail in the middle of sending the files, then you could end up with a situation where files are left on the tablet without database entries. I suppose this is worse than having an updated database where you have all the songs, but not all the files. I can switch this around without too much difficulty.

Selecting "overwrite" for all duplicate files is fine for what you want to do. I'm not sure why it didn't work for you though. I'll have to see what happens if I create some tests to replicate that scenario.

If there are bugs preventing this from working, I'll get them fixed. In the tests I performed earlier, I was able to batch import thousands of files without any problems. What I need to do next is create a scenario where I import a lot of files, then add a ton of files to that directory, and import it again to make sure all the new files are correctly imported. If it all succeeds, then I'll have to go further by trying to create more scenarios where conflicts will occur. I appreciate your patience while we get this all figured out.

Mike

[musichawk]

Mike, a few more facts about the files in MSP's storage location on tablet:

I have confirmed in 3 ways that there are 4941 PDF files on tablet, though the library contains 4222.

My source folder on PC has 4736 files, comprised of 4658 in top level folder plus 78 in one subfolder (used for XMAS songs). Batch Import of this folder does not add more to the tablet.

Looking on the tablet at Date Created of the PDFs, I see a few chunks that match the dates I tried additional Batch Import and tests, but the oldest date is June 15, 9 days ago. If that is accurate, the problem of file- not-in-database happened at least that recently, but probably due to me restoring a backup, which I did around that date or slightly later. The MSP on-tablet Restore process recommends and I always agree to wipe the existing files, then reload them from the backup. If the storage emptying is 100%, the "lost" files got restored from the backup, along with the database that did not have records for all of the files.

Trying to determine when the "extra" (and "lost") files showed up, I just Restored a tablet backup from Jan 24, to see how many files. Result: 4211 files were restored, which matches my current MSP Songs total of 4222 (one extra due to experimenting). Unfortunately, the date of all the restored files is today. But looking at the storage folder, there are 4930 PDF files, even more than before restoring Jan 24 backup. This might make sense because in-between I did some clean-up, removing from the tablet various PDFs that were poor scans, unwanted songs, etc. But of course, the files count as of Jan 24 also greatly exceeds the database count, so apparently whatever sent MSP off the tracks happened months (or years?) before.

Unfortunately, I don't have an older on-tablet backup, Jan 24 is the max of my Way-Back Machine.

But ... I found an older backup yet don't know an effective way to use it for investigation. On my PC I have a older MSPC backup from May 19, 2015. MSPC doesn't offer to Restore this file, so I opened it in View mode to see whats what. Viewing this file in MSPC doesn't give me a count of records nor does it show file dates. I don't know the diff of EXtract Database, Extract, and Extract All, so I tried Extract Database, which dumps out a copy of the .db file.

Then I tried Extract All, which copies all the song files from the backup onto my PC's drive, but resulted in many "Error crating image file", the "Error creating:" seems to be showing Placeholder records, which don't have physical files and/or the Placeholder title text would not be a valid filename. There's no Skip Similar option so I had to work through quite a few of these. (I use Placeholders mainly to have song titles that my band plays from memory, to put them in Setlists, and to remind me to keep looking for or create an actual chart.)

Extract All resulted in a separate folder for each song (which is not how I have set MSP, I don't see a benefit)). Assuming each folder has one PDF, that's 3851, which simply indicates that I added hundreds more songs since May 2016. Folders and files are all dated today. But since I can't seem to Restore this MSPC backup, I can only view and extract it, I don't know how many database records exist.

[Zubersoft]

I am running through more complicated tests, and I'm definitely encountering some problems similar to what you've described. I'm going to keep working through the issues, and I'll let you know when an update is available.

Mike

[Zubersoft]

I was able to track down the root cause of the problems. I'm going to push out updates in the next day or two. Thanks again for your help with this. I'll have you test again after the updates are released.

Thanks,
Mike

UPDATE:
Still fixing a few other unrelated bugs before I push out the update.

[musichawk]

I have the updates installed. Did a new Batch Import, added 765 songs -- yay!

But, I then managed to crash MSPC, found a "trap" that might be something to improve....

In MSPC I wanted to tag all the new imports that came from a just-scanned fakebook with their Album ID, so I searched for that ID which I embed in the file name. About 600 were found (presuming the count, the Search and select actions do not display a count of results, which would be nice).

The I moved too fast -- oops. I selected Edit Songs instead of Batch Edit Songs (adjacent menu items). This resulted in MSPC opening of close to 600 dialogs -- at the same time. Eventually, this crashed it.

Once the rapid series of hundreds of boxes started to open I figured this would not have a good outcome, but there was no evident way to Cancel. That would be an excellent improvement.

And/or, it would be really good for MSPC to realize what it was about to do and ask for verification. Perhaps ask whenever more than maybe 10 records are selected? The app I use for media management (audio, video, images) does this and it is a welcome double-check before craziness happens.

Restarting MSPC, I did the tagging again, properly selecting Batch Edit, and it worked fine.

Thanks for fixing whatever it was that prevented complete importing. Curious, can you say what caused the problem? I could not discern a pattern in which Songs and PDFs were added vs. skipped.

[Zubersoft]

I can look into adding a popup for the scenario when more than 10 are selected. That shouldn't be too difficult. As for the previous fix, it's a little bit hard to explain, but there was an error in the logic when a duplicate file was encountered that caused conflicts with future files to be handled incorrectly.

Thanks,
Mike