Jump to content

Avatar

Songcache Modification Tool


  • Please log in to reply
14 replies to this topic

#1 MFX

MFX
  • 392 posts

Posted October 25, 2016 - 10:33 PM

RB3 Songcache File Scripty Modification Tool Jobber, aka 'CacheTools'

This utility allows you to write custom scripts to modify your RB3 'songcache' file. Typically this would be to accomplish various metadata corrections you might wish to make on a 'batch' basis.

It's also possible to do stranger things with it, but I'll leave that up to your imagination for now.
 
Latest Release: v0.5 alpha (Oct 25, 2016)
 
Download Link

 

Alternate Link

 

This is still alpha software! Only the brave and technically proficient should venture further!
 
Good luck to all who take up the challenge. Hopefully no horrible bugs crept in during my rush to release it. Minimal documentation is found in the following two posts, which are pasted from text files in the zip archive.
 
Have for poking at it and please tell me what you think, both how it works now and what it might become in the future.


(Screenshot of the exhilarating and groundbreaking GUI...)
05RhoXb.jpg



#2 MFX

MFX
  • 392 posts

Posted October 25, 2016 - 10:34 PM

[contents of README.txt file]

 

20161025
v0.5 alpha

RB3 Songcache File Scripty Modification Tool Jobber, aka 'CacheTools'
---------------------------------------------------------------------

This utility allows you to write custom scripts to modify your RB3 'songcache' file. Typically this would be to accomplish various metadata corrections you might wish to make on a 'batch' basis.

It's also possible to do stranger things with it, but I'll leave that up to your imagination for now.


Requirements
------------

I run on Windows 7, so that's all I can guarantee it'll work on.

You'll need some sort of baseline .NET Framework to run this under. At the moment I'm not particularly motivated to figure out which definitive version, but hopefully you can figure that out for yourself. If you can't, then running alpha software probably isn't your bag, and I'd be loathe to have to teach you the particulars of writing scripts.

Also, I only have an Xbox 360, so those are the sorts of cache files I deal with. I build to a CON file that should load onto a USB stick with no issues. If you're on another console, you could still use Nemo's C3CONTools to extract the cache from the CON and do what you will with it.


Installation
------------

Extract the zip archive into a directory. That's pretty much all you need.


Usage
-----

1) Load a songcache file. You should be able to select either a full CON file or an extracted songcache payload.

2) Run a script file. It should tell you if it loaded and/or executed successfully.

2b) If you'd like, run another script file. You can do this as many times as you need, really, but remember that the results are cumulative: if you need a 'reset', just go back to Step 1 and reload the script.

3) Save the modified songcache file. Keep in mind, it should be named 'songcache' by the time you put it on a USB stick to use on the 360.


How to Write Scripts
--------------------

That's in another document.


How to Get Help
---------------

For now, there's a thread on C3 where you can petition the community at large:

http://customscreato...ification-tool/


Legal Crap
----------

USE AT YOUR OWN RISK. I'm not responsible for the trouble you will inevitably cause yourself by mucking about this program.

Do not host this program on your site or any other public archive.

Do not "deep" link the file download directly, as it removes any context for people to get help.

Instead, link to the C3 announcement thread:

http://customscreato...ification-tool/

Do not sell this program in any form, including bundles or archives, virtual or physical.

Otherwise, have fun!  It's free!


Credits
-------

My name is Markleford Friedman, and I take full responsibility for this utility when the jackboots come!

That said, thanks to some folks at C3 are in order: TrojanNemo for sample code and the patched X360 library, pksage for the nascent songcache file format document, and Farottone for being Farottone! :)

- m


History
-------

20161025 v0.5 alpha
- First public release

[end]
 



#3 MFX

MFX
  • 392 posts

Posted October 25, 2016 - 10:35 PM

[contents of SCRIPTING-HOWTO.txt file]

 

DESIGN PHILOSOPHY
-----------------

I had my own needs for fixing song metadata in the cache, but I figured making something that would satisfy all comers might be cool.

But hey, for all I know nobody else is gonna care to use it! So I may as well have some fun toward my ends, and that included taking on a programming challenge since I've let my skills atrophy while I attempt to write novels and produce an album for the past three years instead of holding down a day job.

The project direction pretty much came down to the choice of "Script-like or SQL-like"? While the songcache could certainly be seen as a sort of database to run structured queries against, I eventually figured that writing a scripting grammar would provide better experience to leverage in future projects.

So a scripting system it is. Simple for now: no symbols or extended expressions, no user-declared variables, and a limited number of attributes to get and set.


GRAMMAR
-------

Okay, so I forgot the right notation for a proper BNF (comp-sci degree 24 years old now), but this is what I came up with when designing the script grammar...


SCRIPT :: BLOCK

BLOCK :: LINE *

LINE :: <BRANCH | SELECT | COMMAND | BLANK> [ REM ]

BLANK :: (WHITESPACE)

REM :: ; (TEXT)

STRING :: "(TEXT)"

ATTRIBUTE :: song[name] | id | shortname | artist | album | yearrecorded | yearreleased | vocalparts | gameorigin | genre | rating | gender

COMMAND :: set ATTRIBUTE to EXPRESSION | using ATTRIBUTE replace STRING with EXPRESSION | print[ln] EXPRESSION

SELECT :: each SELECTOR do BLOCK end

SELECTOR :: ATTRIBUTE within SET

SET :: <STRING | INT> *

BRANCH ::  <if | once> CONDITION <then | do> BLOCK [elsif CONDITION then BLOCK] * [otherwise BLOCK] end

CONDITION ::  CLAUSE [ and CLAUSE | or CLAUSE ] *

CLAUSE :: COMPARATOR | SELECTOR

COMPARATOR :: ATTIRIBUTE [eq | ne | contains] EXPRESSION

EXPRESSION :: <ATTRIBUTE | STRING | INT> [ + EXPRESSION ]


QUIRKS and CAVEATS
------------------

Capitalization of keywords doesn't matter.

To print a quotation mark in a string, escape it with a backslash: "Here are some \"quotes\"."

For now, there are no proper expressions. They're merely string concatentations. (To that end, you actually don't need a '+' to connect them, currently, but I imagine you'll want to use them in case that changes in the future.)

To that end, integers are compared as strings. Everything is stringified for comparing! When you want to set an attribute with a numeric value, though, I attempt to parse back to integer: if that fails, then no value is set.

Furthermore, there is no range/value validation for integer attributes. Set them at your own risk.

Chaining if/branch conditions with 'and'/'or' is pretty much only meaningful if you don't mix conjunctions, since we currently don't have expressions advanced enough to allow parentheses for 'order of operation'.


EXECUTION, or HOW THESE THINGS ACTUALLY WORK
--------------------------------------------

The program iterates over every song found in the songcache. Note that this includes any song that your console ever saw since the last full cache build, so there could be some crap customs still in there that you deleted ASAP after testing them (aherm, I'm looking at you, NON-PITCHED VOCAL SONGS!).

For each song, it runs the selected script in its entirety, top to bottom. During this execution, the named attributes only apply to the current song scoped by the iterator.

This, however, means there could be a bit of wasted CPU cycles in comparing the same conditions over and over (especially string matching). Not that it takes more than a fraction of a second to run a script on a songcache of moderate size (unless you use print/ln a lot: that is quite costly!), but you can use the 'once' and 'each' branch types to help with this: once their conditions pass as 'true', they're short-circuited to always returning 'false' and are never checked again.


FINAL ADVICE
------------

ALWAYS keep a backup of your cache before poking at it!

You WILL mess it up at some point and will want to roll back!


EXAMPLES
--------

In my experience, having a half-assed BNF to look at is nice, but you really learn the most from examples.

Here are some snippets to learn from:
 

; 'Comments' start with a semicolon
; They can be on lines by themselves


; Or a comment can follow a line
; Everything on a line after the semicolon is ignored
println song   ; Printing out song name


; Strings are in double quotes
if name eq "Proud Mary" then
    set vocalParts to 3
end


; Create an expression at any time by concatenating strings
println "Parsing song " + id + " (" + shortname + ")"


; Ids are numeric
if id eq 1006117 then
    set vocalParts to 3
end


; contains = search within a string
; Label all songs charted by Farottone
if shortname contains "far_" then
    set name to name + " (FAR!)"
end


; For now, we allow 'contains' with numbers
; Here, non wipe-proof song IDs always start with 10746
if id contains 10746 then
    println name + " not wipe-proof"
end


; Nested ifs, of course
if artist eq "David Bowie" then
    if album eq "Lets Dance" then
        set album to "Let's Dance"
    end
end


; Chaining if conditions, doing the same thing
if artist eq "David Bowie"
and album eq "Lets Dance" then
    set album to "Let's Dance"
end


; The OR case
if artist eq "Queensryche"
or artist eq "Queensreich" then
    set artist to "Queensrÿche"
end


; 'once' = like an if, but after the first time it matches we never look for it again (efficiency)
once shortname eq "proudmary2" do
    set vocalParts to 3
end


; An extended chain:
; Use as name elsif's as needed,
; End with an 'otherwise' as catch-all case
if artist contains "Creedence" then
    if name eq "Proud Mary" then
        set name to "Proud Mary" + " (with Harmonies)"
    elsif name eq "Fortunate Son (Original Version)" then
        set name to "Fortunate Son (Not a Lame Cover)"
    otherwise
        set name to name + " (CCR roolz!)"
    end
end


; Matching a string, then replacing it
if name contains " (Original Version)" then
    using name replace " (Original Version)" with "" ; Use the null string ("") to delete substr
end


; Matching a set
if artist within "Muse" "Rush" "David Bowie" then
    ; Prefix released year on these albums to be able to sort chronologically
    set album to released + ". " + album
end


; 'each' = like matching a set, but only once for each: when we find it, remove from list
; Once every member is found, we skip the entire check
each shortname within
    "badmoonrising" "fortunateson2" "downonthecorner" "whollstoptherain" "proudmary2"
do
    set artist to "Creedence Clearwater Revival (CCR)"
end



#4 TrojanNemo

TrojanNemo

    The Programmer

  • 4,872 posts
  • Location
    Miami, Florida

Posted October 26, 2016 - 01:06 AM

First!

#5 MFX

MFX
  • 392 posts

Posted October 26, 2016 - 01:28 AM

First!

 

Someone ban this joker! :argh:



#6 TrojanNemo

TrojanNemo

    The Programmer

  • 4,872 posts
  • Location
    Miami, Florida

Posted October 26, 2016 - 07:37 PM

Just saw you updated the posts. Read through it all, good job. Sounds very cool. You need a screenshot in the OP ;-)

BAM!

05RhoXb.jpg

#7 MFX

MFX
  • 392 posts

Posted October 27, 2016 - 01:04 AM

BAM, indeed. Attached to OP.

Thanks for the screenshot: now people are sure to want to use it! ;)

#8 TrojanNemo

TrojanNemo

    The Programmer

  • 4,872 posts
  • Location
    Miami, Florida

Posted October 27, 2016 - 01:10 AM

It looks nice and seems to work fine on the surface. I unfortunately don't have time to actually put it to the test, but i'm sure others will. I'm glad I was able to help get you going on the right direction.

#9 Bansheeflyer

Bansheeflyer

    Moderator

  • 1,379 posts
  • Location
    The Mothball Fleet

Posted October 27, 2016 - 01:46 AM

Awesome program! Nice job.


As a moderator I enforce the law...in moderation.

 

My Customs
My Works in Progress


#10 Farottone

Farottone

    The Lean Mean Customs Charting Machine

  • Administrator
  • 6,411 posts

Posted October 27, 2016 - 09:29 AM

Amazing piece of software man. :)



#11 MFX

MFX
  • 392 posts

Posted October 27, 2016 - 04:28 PM

Awesome program! Nice job.

 

Amazing piece of software man. :)

 

Thanks, peoples! :)

 

But answer me this: Is the concept at all useful? It certainly makes for a fun little curiosity, but given that people can already make changes in the dta (made even easier by Nemo's Quick Editor), even I come up short for identifying a 'big win' use case for doing it programmatically, particularly since the changes get wiped out if your cache is rebuilt.

 

The closest I've come to "uniquely powerful" is a yet-unreleased system to consume a .csv file of song reviews from both the missus and me that turns them into a cross-referenced series of custom genres. In this manner, it's quite easy for us to filter all of the songs that we've ranked 4 or 5 lighters, which makes the 'party shuffle' feature actually useful for a change.

 

Also along those lines, I use those .csv reviews to make it look like the songs that both of us have rated 2 or 1 have Pro Guitar support, which means all we have to do is tick the 'Pro Guitar NO' filter and instantly all the crap from exports is filtered out, yet is still available if we have party guests that like the crap. ;) (Mind, it works for us because we don't have a Pro Guitar.)

 

Both of those experiments, though, have too many moving parts for public release at this point.

 

So part of the reason for releasing the scripting system to the world is to see if other people can make something useful of it. To that end, if you manage to come up with something that's useful to you (beyond what you can do with Quick DTA Editor), please share your little victories! :)



#12 Farottone

Farottone

    The Lean Mean Customs Charting Machine

  • Administrator
  • 6,411 posts

Posted October 27, 2016 - 04:43 PM

I'm gonna be honest and say I'm not gonna use it because it's not what I would usually do. However, the same can be said for a particle collider, it's still amazing even if I don't use it. :)



#13 TrojanNemo

TrojanNemo

    The Programmer

  • 4,872 posts
  • Location
    Miami, Florida

Posted October 27, 2016 - 04:46 PM

It's definitely cool and has great potential...but for me at least, it's 2+ years too late. I haven't fired up my Xbox 360 in, what, a year? If I ever play RB3 again I certainly won't mess around with things. Unfortunately for those of us who have created tools for this community (now that includes you), there's little activity going on and there's no guarantee that anything you put out will get a ton of use. It is likely, however, that one or two people will use it and think it's the best thing in the world. But those will be limited cases. For me, for example, it's a mystery that cPlayer isn't much more popular. I think it's so great at what it does. But it also probably came out too late.

#14 MFX

MFX
  • 392 posts

Posted October 27, 2016 - 09:08 PM

Yeah, even if I managed to launch this before the tail-end of the scene, it's an "esoteric" idea to begin with! That's why I need help in figuring out whether it has a niche that I can't quite see yet, since even I'm coming up short for practical uses for it. I wasn't thinking "practical utility" when I decided to do a scripting engine, just "challenging exercise": my particle collider, if you will... ;)

 

Oh, I forgot one other thing that I use it for, which was actually included as a script example:

if artist within "Muse" "Rush" "David Bowie" then
    ; Prefix released year on these albums to be able to sort chronologically
    set album to released + ". " + album
end

I hate the way RB3 sorts albums alphabetically, so this was my fix for bands with a lot of customs. It's actually something that it's quite suited for, rather than going into all the DTAs and fixing by hand.

 

Actually, most of my cache experiments have been of the sort of "HMX really dropped the ball on filtering and sorting, let's see how we can work around it". Seriously, why didn't they pay more attention to allowing players to "curate" their collections? Why in the world would they stop at the TU5 feature of "don't show 1-lighter songs" instead of providing an actual filter for "review"? Why do playlists seem so tack-on?

 

The mind boggles, at times, why such low-hanging fruit slips through their fingers, and from what I've seen from RB4 they've yet to improve on library management much at all.



#15 MFX

MFX
  • 392 posts

Posted September 5, 2018 - 01:24 AM

Been away from here forever, but I got a message that the KeepItFishy link is down.

 

I've now put it up for download here also:

 

Alternate Link






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users