08-07-2023, 11:25 PM (This post was last modified: 08-07-2023, 11:32 PM by Jeffrocchio@gmail.com.)
Creating a thread to openly discuss a process for migrating from forScore to MobileSheets. I personally have about 1,000 scores in forScore that I will be migrating over to MobileSheets. My goal is to do this while retaining as much of the metadata I have in forScore as possible. Any annotations in forScore will, of course, be lost. I will note that in my case I have only a few of those so for me this is not an issue.
My first draft 'sketch' of the process is:
* Export all scores from forScore.
* Export metadata from forScore into a CSV file.
* Modify CSV file for use with exiftool to match what MobileSheets expects.
* Use modified CSV file with exiftool to populate all metadata into the individual PDF docs (as a single batch command in exiftool).
* Import all the PDF docs into MobileSheets as a batch command (or perhaps in separate batches based on setlists).
Current Unknowns:
1. Can I fully match all forScore metadata fields and content with those in MobileSheets? (E.g., 'Difficulty,' 'Tempo,' and such.) <- Confidence Level=High.
2. Can all metadata fields in MobileSheets be 'auto-populated' from fields in the imported PDF doc? (E.g., 'Keys,' 'Difficulty,' and such).
3. Is there some way to migrate over setlists as well? <- Confidence Level=Low.
4. Is there some way to migrate over forScore 'bookmarks'? These are sets of pages within a large PDF document that are treated in forScore as an individual score (including the ability to set unique metadata for the bookmarked page-set). The typical use-case being a book of songs where you've 'bookmarked' 1-2 pages that is one of the songs in the book. I know MobileSheets can do this as well; just don't know how I'd migrate over that internal information. However, the forScore metadata export CSV file does contain the information that would be needed to do it.
5. Can the CSV file modifications be scripted to make this step more reliable for others wishing to use this process?
Question for Mike: Can you answer question #2 above? (Note: I did a quick test to try to populate the 'Keys' field in MobileSheets, but it did not auto-populate upon import. I just created a field in a PDF doc as "Keys=D." But maybe the content has to be a number or some other way that you're encoding the Keys field internally?)
08-08-2023, 12:37 AM (This post was last modified: 08-08-2023, 12:47 AM by BRX.)
Are you aware that you don't need to populate the PDF with metadata and use exiftool to get your data into MS?
If you want to have metadata saved in the PDF, then OK. But you can import straight with the csv and the pdf to get the metadata into the MS database and use it to catalog your music.
I don't use metadata saved in the PDFs themselves at all in my > 55.000 song database. That might only be relevant if you want to use the PDFs outside of MS for something, but as soon as you have the metadata in the database of MS you don't neet "hardcoded" metadata in the PDFs from within MS.
Just adjust your exported csv file to the columns and format MS needs and you can import.
P.S. I usually think about importing from a pdf in terms of fakebooks and other collections as source. Importing of single songs/files is a bit tedious as long as Mike hasn't changed the import process to be able to do this in batch processing as well. But I guess the current behaviour (extracting single songs and page ranges from a volume) is what you are looking for in your point 4.
I better let Mike answer the rest of your questions. I haven't imported songs from csv for a long time. But from that another advice: Keep metadata information that isn't compatible with MS yet in a big source csv or spreadsheet file just for later when it will be compatible in the future. I've been waiting for the addition of some additional metadata columns for quite some time for which I have the information ready in my "master spreadsheet" to import when it's possible.
BRX: thanks for your reply. However, where you say >you can import straight with the csv and the pdf to get the metadata into the MS database < -- in the thread "Questions from long-time forScore user", on 07-31-2023, 05:50 PM Mike wrote: "The CSV feature is currently only meant to be used to extract songs from a larger PDF (it acts as an index of the available songs in the PDF). I do not currently support importing a CSV to populate metadata for a large number of songs/files at once."
Also note that MobileSheets will do batch import of PDF docs, and auto-populate the metadata from the PDF as it imports the files. I have tested this feature by importing 100 scores from a directory on my Dropbox account and it worked. From the currently published user manual: "There is also a batch import feature that can import all files from a selected directory and sub-directories"
If you have used your CSV doc to populate metadata for 100s-1000s of songs in MobileSheets - can you advise on the steps for doing that?
>You mentioned "Keys=D." - have you tried "Keys=D" ? <-- Good thought, but the period was just in this post, not in the actual metadata. I also tried with naming the field in the PDF 'keys' - lowercase. That doesn't work either.
>I think you will need to have a 2 line csv file for each song being input (first line denoting fields, 2nd line the entries for the song) <-- Ok. However, for me this would be lot more work than using the one-line-per song, single CSV file, forScore library export as the input to exiftool to batch populate into each PDF doc. Depending on Mike's answer to my question #2 that is, I suppose. And there is also some value to having the metadata travel with the PDF doc itself. Partly because I create some of the scores myself and they will be used across both MobileSheets and forScore. And partly because it fits my workflow. Tho a consolidated database in a worksheet readily available on my desktop PC would be handy....
(08-08-2023, 12:37 AM)BRX Wrote: you can import straight with the csv and the pdf...
BRX: You did, with the above statement, point me to the likely solution to my "Unknown #4." In looking at the User Manual I see a section titled: "CSV OR PDF BOOKMARK IMPORT." This is the 2-line CSV file you are talking about; and it's purpose is actually to achieve specifically my item #4. Tho I will still need to work out the most efficient way to get from the info in the forScore exported CSV into multiple, separate, CSVs for MobileSheets import.
Not a great deal of work if you create a program to generate the 2 line csv's from your master set of metadata.
You can probably batch import multiple files at the same time - just put all the songs and their 2 line csv's into a folder and batch import from that.
Suggest you try it with 2 manually created csvs to prove that it picks up both songs with a single import.
(08-07-2023, 11:25 PM)Jeffrocchio@gmail.com Wrote: 3. Is there some way to migrate over setlists as well? <- Confidence Level=Low.
The User Manual section titled: "Populating Metadata from Directories" may contain the answer to "Unknown #3." That is, using subdirectories based on setlists with the batch import feature.
OR by making use of the .mss/XML file export option: "MobileSheets also introduces a file format called .mss (MobileSheets Setlist Songs) which is XML based and can be viewed/edited in a text editor if needed. This file format contains just a list of songs in the setlist along with information about the file each song uses. When this file is imported on another device, it will try to create the same setlist with matching songs found in its library. If no matching song is found, it will be skipped and not added to the setlist."
08-08-2023, 03:49 AM (This post was last modified: 08-08-2023, 08:59 AM by Zubersoft.)
1. Can I fully match all forScore metadata fields and content with those in MobileSheets? (E.g., 'Difficulty,' 'Tempo,' and such.) <- Confidence Level=High.
Yes, I believe so. I don't know how much freedom forScore allows with creating custom fields. I am planning on expanding the number of custom groups and fields with the next major update.
2. Can all metadata fields in MobileSheets be 'auto-populated' from fields in the imported PDF doc? (E.g., 'Keys,' 'Difficulty,' and such).
Not every field, no. I'm currently looking for all of the group types in the PDF metadata (Artist, Album, Composer, Genre, Key, Collection, Setlist, Signature, CustomGroup, Year, SourceType), as well as Duration and Keywords. No other fields are extracted from PDF metadata currently.
You can certainly use the batch import with directories to populate metadata. The problem with csv files is that you can't batch import them, so you'd have to do it one song at a time. I am going to try to add support for importing multiple files at once through csv in one of the next major updates. Unlike the current csv import feature, you won't pick which songs to create - it will just create one song for every file found in the csv.
The last thing I should mention is that I'm going to try to add basic support for importing forScore 4SC and backup files. Over time I may be able to extract more information, but my shorter term goal is to at least be able to extract all the metadata and PDFs. This may include Setlists - it will just depend on how difficult it is to parse through the files. I'm going to have to spend some time figuring out the file formats for this.
3. Is there some way to migrate over setlists as well? <- Confidence Level=Low.
.mss files could certainly work for that. It's also supported with PDF metadata, or when using the batch import feature with the option to populate metadata from directories. Or as mentioned above, if I can figure out the file formats, that may be a more direct way, but that's going to take me a while with all of the other ongoing work, so it may be faster for you to figure out a different solution in the meantime.
4. Is there some way to migrate over forScore 'bookmarks'? These are sets of pages within a large PDF document that are treated in forScore as an individual score (including the ability to set unique metadata for the bookmarked page-set). The typical use-case being a book of songs where you've 'bookmarked' 1-2 pages that is one of the songs in the book. I know MobileSheets can do this as well; just don't know how I'd migrate over that internal information. However, the forScore metadata export CSV file does contain the information that would be needed to do it.
This is one of the biggest incompatibilities. The problem is that bookmarks only link to a single page in MobileSheets right now instead of a range of pages. This means they can't be used to define a psuedo-song like you can in forScore. That is something I will probably address at some point, but it requires some pretty significant design changes, and I'm not sure how far I want to go down the rabbit hole with this. I have the "Create Snippet" feature in MobileSheets for this purpose where a separate song is created in the library from a range of pages in the original PDF. This serves basically the same purpose, but I'm not sure how this differs from the bookmarks approach in forScore, and what benefits using bookmarks has over this.
5. Can the CSV file modifications be scripted to make this step more reliable for others wishing to use this process?
I'm not sure if you are asking for changes from me with this, or if it's just a general statement for anyone interested in helping write scripts to generate CSV files to move the data over.
Question for Mike: Can you answer question #2 above? (Note: I did a quick test to try to populate the 'Keys' field in MobileSheets, but it did not auto-populate upon import. I just created a field in a PDF doc as "Keys=D." But maybe the content has to be a number or some other way that you're encoding the Keys field internally?)
I'm looking for "Key" at the moment (I don't pluralize any of the group fields when looking for metadata, so it's Genre, Key, Year, etc). I believe you need to add a field like, <xmp:Key>C</xmp:Key> which is under the rdf: Description element in the metadata. Let me know if you need more information on this.
(08-08-2023, 03:49 AM)Zubersoft Wrote: Mike provided a detailed response to my original post. Below is my reaction to his responses, quoting the text in Mike's response in >brackets<....
Excellent info Mike. I'll need to digest it all; but on the surface I think I'll be able to achieve a good 85-90% of my migration with this info and the update you mentioned in the Issue: Albums Metadata Field as List on PDF Import thread, to wit: "I think what I will do is separate based on either comma or semi-colon for all fields except Artist and Composer. ..."
Specifics:
>I don't know how much freedom forScore allows with creating custom fields <-- None at present. forScore only allows for renaming fields, not adding new ones.
>2. Can all metadata fields in MobileSheets be 'auto-populated' from fields in the imported PDF doc? ... I'm currently looking for...Artist, Album, Composer, Genre, Key, Collection, Setlist, Signature, CustomGroup, Year, SourceType, Duration, and Keywords. <-- Got it. The only forScore related fields missing are Rating and Difficulty then. I hardly ever use those, so I'm good to go with this set of auto-populated fields.
>3. Is there some way to migrate over setlists as well? ... .mss files could certainly work for that. It's also supported with PDF metadata, or when using the batch import feature with the option to populate metadata from directories. <-- Yes, good. This gives me several options to consider.
>4. Is there some way to migrate over forScore 'bookmarks'? I have the "Create Snippet" feature in MobileSheets for this purpose where a separate song is created in the library from a range of pages in the original PDF. This serves basically the same purpose, but I'm not sure how this differs from the bookmarks approach in forScore, <-- Not having played with MobileSheets Snippets I can't say for sure; but on the face of it your description sounds like it is really the same thing. There might not be any benefit to adding 'bookmarks' over Snippets.
>5. Can the CSV file modifications be scripted to make this step more reliable for others wishing to use this process? <-- This was just for me and any others interested. I get by in C++ for Auduino and Raspberry Pi; but less so in the various scripting languages that might make sense here. And the CSV file mods may be easy enough with some formulas and a bit of search-and-replace.
>I'm looking for "Key" at the moment (I don't pluralize any of the group fields when looking for metadata, <-- Got it, cool. Yes, for exiftool I have a .ExifTool_config file where I define the needed PDF metadata fields not 'native' to the Adobe PDF spec. Another user pointed me in the right direction and I now have 'Key' working fine.
I have also attached a worksheet that contains a field mapping between forScore, PDF and MobileSheets - as I currently understand it. This will may evolve a bit still as I work through my migration process. See the sheet named forScore Field Mapping.
08-10-2023, 11:26 PM (This post was last modified: 08-11-2023, 12:18 AM by Jeffrocchio@gmail.com.)
(08-08-2023, 03:49 AM)Zubersoft Wrote: ... I'm currently looking for all of the group types in the PDF metadata ..., SourceType ...as well as Duration...
Mike: I have not been able to successfully import either the Duration field or the Source [SourceType | Sources] field.
For Duration I have tried it as both text in the form "01:05" and as just the number of seconds as "65." In my test PDF doc this latter form shows as:
[XMP] Duration: 65
For SourceType I have tried all variants I can thing of: Source / Sources / SourceType / Sourcetype. Can't get any of those to work.
08-11-2023, 03:57 AM (This post was last modified: 08-11-2023, 04:03 AM by Zubersoft.)
The code is currently expecting the duration in milliseconds. So you can enter 65000 for 65 seconds. I will change this to seconds - it's truncating the result to seconds by dividing anyways, so there isn't much benefit to using milliseconds.
As far as SourceType, I look for it in the same manner as every other field, and will accept sourceType or SourceType. I'm really not sure why that would not be working for you when you used SourceType though, as the code is written identically for that field as the others.
(08-11-2023, 03:57 AM)Zubersoft Wrote: The code is currently expecting the duration in milliseconds. So you can enter 65000 for 65 seconds. ...
As far as SourceType, I look for it in the same manner as every other field, and will accept sourceType or SourceType. I'm really not sure why that would not be working for you when you used SourceType though, as the code is written identically for that field as the others.
Duration: I validated that 65000 works.
SourceType. I posted this issue to the ExifTool forum this morning, with my code. The tools author, Phil Harvey, stated that the field's "Description" is showing as "Source Type" <-- with a space in the name. But the Description is not the actual field name, which is correctly showing as SourceType (in the example I provided to him). Is there any chance your code is inadvertently using the Description vs the actual name? I'm just taking a shot in the dark here.
I've attached the offending PDF if you want to inspect it.
