Bf4 Sbtoc Dumper

[Frankelstner]

Grab the latest version of the script here: https://www.dropbox.com/s/rhu9gjxs9087vn7/bf4dumper.zip

Description of the LZ77 compression: http://pastebin.com/u2kntxSV

Description of the changes to the sbtoc: http://pastebin.com/TftZEU9q

Documentation of what I did to figure out the compression in case you are interested: http://pastebin.com/rGpBFwAV

Documentation of what I did to figure out the sbtoc changes: http://pastebin.com/0bZebD8S

Update 8.1.2014: Added support for the noncas archives (both unpatched and patched). I've rewritten most of the scripts along the way.

Update 17.1.2014: Fixed handling of noncas delta payload instructions: 1) Deltas without implicit base-reading section at the end are now handled correctly. 2) Fixed the sanity check for instruction type 3.

Update 1.2.2014: Noncas deltas ending with instruction type 4 are now handled correctly.

Edited February 1, 2014 by Frankelstner

[Durandal-217]

You the boss, you know that right?

Thanks for all your hard work, it is very much appreciated, will report back if there are any problems.

[Durandal-217]

n/m I figured it out I wasn't putting the right path, anyway I got it now.

Edited November 7, 2013 by Durandal-217

[nightmaremutant]

good work man, gonna try it out when i get home, just a question, will the audio decoder work on the files this dumper makes or do we have to wait a bit longer for you to make the decoder work with this?

[Durandal-217]

good work man, gonna try it out when i get home, just a question, will the audio decoder work on the files this dumper makes or do we have to wait a bit longer for you to make the decoder work with this?

Nope don't think so, on the beta files I keep getting this error.

Traceback (most recent call last):
 File "C:\Users\Durandal-217\Desktop\fb2audio\fb2decoder.py", line 624, in <module>
   decodeAudio()
 File "C:\Users\Durandal-217\Desktop\fb2audio\fb2decoder.py", line 65, in decodeAudio
   dbx=Dbx(f,unpack)
 File "C:\Users\Durandal-217\Desktop\fb2audio\fb2decoder.py", line 254, in __init__
   self.fieldDescriptors=[FieldDescriptor(self.unpack("IHHII",f.read(16)), self.keywordDict) for i in xrange(self.header.numField)]
 File "C:\Users\Durandal-217\Desktop\fb2audio\fb2decoder.py", line 109, in __init__
   self.name            = keywordDict[varList[0]]
KeyError: 1634562671
>>> 

I can't confirm if this happens with retail files.

[Frankelstner]

The decoder uses an ebx parser to grab the filenames. Obviously the script cannot magically know about the new ebx format. Will try to adjust it now, which should take just an hour or two if no other changes were made to the actual audio chunks. Though as you may imagine, I'm not sure what the chances are that the chunks were not modified as well.

[-=anders=-]

Strange unlogical error to me with the dumper...

I have these paths:

##Adjust paths here. The script doesn't overwrite existing files so set tocRoot to the patched files first,
##then run the script again with the unpatched ones to get all files at their most recent version.

bf4Directory=r"F:\Spel\Battlefield 4"
outputfolder="H:\Ljudeffekter\BF3 audio\bf4 dump"



catName=bf4Directory+"F:\Spel\Battlefield 4\Data\cas.cat" #use "" or r"" if you have no cat; doing so will make the script ignore patchedCatName too
patchedCatName=bf4Directory+"F:\Spel\Battlefield 4\Update\Patch\Data\cas.cat" #used only when tocRoot contains "Update"

tocRoot=bf4Directory+"F:\Spel\Battlefield 4\Update\Patch\Data\Win32"
##tocRoot=bf4Directory+"\\"+r"Data\Win32"

Get error message, telling me:

Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
>>> 

Traceback (most recent call last):
 File "H:\Ljudeffekter\BF3 audio\dumper.py", line 401, in <module>
   cat=Cat(catName)
 File "H:\Ljudeffekter\BF3 audio\dumper.py", line 146, in __init__
   cat2=open(catname,"rb")
IOError: [Errno 22] invalid mode ('rb') or filename: 'F:\\Spel\\Battlefield 4F:\\Spel\\Battlefield 4\\Data\\cas.cat'
>>> 

[Frankelstner]

Actually you only need to change bf4Directory and outputFolder.

[-=anders=-]

whoops. Sorry... Now let's see. ;)/> thx

Big thx for this mate!

Edited November 8, 2013 by -=anders=-

[-=anders=-]

Jesus! Have been dumping the "update" files for 1 hour now.

[Durandal-217]

Jesus! :D/> Have been dumping the "update" files for 1 hour now. :)/>

It took me over 14 hours to dump the full game.

[-=anders=-]

Yeah, well Im done now.. Didnt take 14 hours though.. Maybe 3..

[Durandal-217]

Yeah, well Im done now.. :D/> Didnt take 14 hours though.. Maybe 3.. :)/>

That's strange, my computer isn't all that old or slow, it dumped BF3's files in about an hour or more, but for BF4 it took 14 hours. I know frank said the script was slow, but for me that was just too slow.

[Frankelstner]

That's strange, my computer isn't all that old or slow, it dumped BF3's files in about an hour or more, but for BF4 it took 14 hours. I know frank said the script was slow, but for me that was just too slow.

I think anders dumped the beta files, not the full game. 14 hours seems like a good estimate.

I'm trying to improve the decompression (which is the bottleneck). With a small sample of files, the version here takes about 85 seconds. I've cut that down to about 25. I suppose when written in C I might get this down to 15 so most time is spent reading and writing the files and not decompressing.

[-=anders=-]

I dumped the full version game...

When looking over the EBX files etc. I discover LOTS of sounds etc.

So first I did as you recommended to dump the patched files first like this:

##Adjust paths here. The script doesn't overwrite existing files so set tocRoot to the patched files first,
##then run the script again with the unpatched ones to get all files at their most recent version.

bf4Directory=r"F:\Spel\Battlefield 4"
outputfolder="H:\Ljudeffekter\BF3 audio\dump5"



catName=bf4Directory+"\\"+r"Data\cas.cat" #use "" or r"" if you have no cat; doing so will make the script ignore patchedCatName too
patchedCatName=bf4Directory+"\\"+r"Update\Patch\Data\cas.cat" #used only when tocRoot contains "Update"

tocRoot=bf4Directory+"\\"+r"Update\Patch\Data\Win32"
##tocRoot=bf4Directory+"\\"+r"Data\Win32"

Then I dumped the non patched files like this:

##Adjust paths here. The script doesn't overwrite existing files so set tocRoot to the patched files first,
##then run the script again with the unpatched ones to get all files at their most recent version.

bf4Directory=r"F:\Spel\Battlefield 4"
outputfolder="H:\Ljudeffekter\BF3 audio\dump5"



catName=bf4Directory+"\\"+r"Data\cas.cat" #use "" or r"" if you have no cat; doing so will make the script ignore patchedCatName too
patchedCatName=bf4Directory+"\\"+r"Update\Patch\Data\cas.cat" #used only when tocRoot contains "Update"

tocRoot=bf4Directory+"\\"+r"Data\Win32"
##tocRoot=bf4Directory+"\\"+r"Data\Win32"

Had an error with the non patched files and with the script like this aswell ^^.

But when using the decoder, there are sounds that are cut offed and the sounds are FAR from beeing decoded. Like 20% are decoded...

I had more decoded sounds when I decoded the beta... So something when really wrong for me. :(/>/>

Edited November 8, 2013 by -=anders=-

[Frankelstner]

I've rewritten the decompression and patching function in C++. Took me 37 minutes to dump the entire game. However, the script now requires a dll to do anything at all.

[rukqoa]

Nice! I was wondering where people were getting the ingame data.

Out of curiosity, is the compression (reverse) process trivial busywork or is it different?

[Frankelstner]

Compressing is a bit more complicated as it requires looking into the past of the file and finding matches that can be replaced. It's not that hard though.

However, it is not necessary to actually compress the files again. Each block specifies at the beginning whether it is compressed or uncompressed. Just using uncompressed data throughout the file with a little overhead (8 bytes for every 0x10000 bytes payload) is the most viable option. In fact I've done that when adapting the file tweaker to MOHW (which uses a similar format but with zlib). That one is somewhere on mordor and still working I believe (ugh, weapon damage mods; thankfully nobody cares).

[-=anders=-]

So this is what happened this time... Nothing I recognized and it took like 1 minute for both patched and original files:

>>> 
CommonChunks0.toc
CommonChunks1.toc
FrontEndChunks.toc
Globals.toc
MpCharacter.toc
MpChunks0.toc
MpChunks1.toc
MpVehicles.toc
Persistence.toc
PrologueChunks.toc
SpChunks0.toc
SpChunks1.toc
UI.toc
WeaponChunks.toc
WeaponsAndAttachments.toc
FrontEnd.toc
MP_Abandoned.toc
MP_Damage.toc
MP_Flooded.toc
MP_Journey.toc
MP_Naval.toc
MP_Playground.toc
MP_Prison.toc
MP_Resort.toc
MP_Siege.toc
MP_TheDish.toc
MP_Tremors.toc
SP_Airfield.toc
SP_Dam.toc
SP_Naval.toc
SP_Prison.toc
SP_Prologue.toc
SP_Shanghai.toc
SP_Suez.toc
en.toc
en_CommonVO.toc
en_MpVO.toc
en_SpVO.toc
>>> 

Only toc files? I dont get it.

[Vosvoy]

Many thanks Frankelstner you're definitly our lil' hero.

[ssbeye]

Hi! Can you reupload LZ77 lib for python script?? Looks it removed(((

[Frankelstner]

Hi! Can you reupload LZ77 lib for python script?? Looks it removed(((

Okay here: http://i.imgur.com/wd0QYet.png Save on your computer, open in paint, then save as 24bit bmp, open in text or hex editor and remove everything before PK, rename to .zip and open it.

[ssbeye]

Okay here: http://i.imgur.com/wd0QYet.png Save on your computer, open in paint, then save as 24bit bmp, open in text or hex editor and remove everything before PK, rename to .zip and open it.

Thanks!!! But easier just open using 7-zip... without any inside mmanipulaitons )))

[ssbeye]

But getting errors:

Traceback (most recent call last):

File "F:\Development\Ninja_Ripper\Battlefield 4 dumper\dumper.py", line 13, in <module>

LZ77 = cdll.LoadLibrary("LZ77")

File "C:\Python27\lib\ctypes\__init__.py", line 431, in LoadLibrary

return self._dlltype(name)

File "C:\Python27\lib\ctypes\__init__.py", line 353, in __init__

self._handle = _dlopen(self._name, mode)

WindowsError: [Error 193]

[ssbeye]

Ok I get ... i just update from Python 2.7.2 to 2.7.6 and all is ok )))