Jump to content
Official BF Editor Forums
Frankelstner

File Dumper For Sbtoc Archives

Recommended Posts

Thanks for the funny swf cutscene thing!

Anyway, I tried running the script for these update files and it popped another error, after 7002 files:

...

Thanks again for the quick and comprensive replies!

Ah, zlib once again. I've used a heuristic approach to determine whether a file has this Frostbite specific zlib format or if it is just uncompressed data. The script apparently made a wrong choice somewhere. Now, I've toyed with the function a bit more and it *should* work better than before.

Replace the zlibb function in dumper.py; use the one below:

def zlibb(f, size):
   #if the entire file is < 10 bytes, it must be non zlib
   if size<10: return f.read(size)

   #interpret the first 10 bytes as fb2 zlib stuff
   uncompressedSize,compressedSize=unpack(">ii",f.read(8))
   magic=f.read(2)
   f.seek(-10,1)

   #sanity check: compressedSize may be just random non-zlib payload.
   if compressedSize>size-8: return f.read(size)
   if compressedSize<=0 or uncompressedSize<=0: return f.read(size)

   #another sanity check with a very specific condition:
   #when uncompressedSize is different from compressedSize, then having a non-zlib piece makes no sense.
   #alternatively one could just let the zlib module try to handle this.
   #It's tempting to compare uncompressedSize<compressedSize, but there are indeed cases when
   #the uncompressed payload is smaller than the compressed one.
   if uncompressedSize!=compressedSize and magic!="\x78\xda":
       return f.read(size)

   outStream=StringIO()
   pos0=f.tell()
   while f.tell()<pos0+size-8:
       uncompressedSize,compressedSize=unpack(">ii",f.read(8)) #big endian

       #sanity checks:
       #The sizes may be just random non-zlib payload; as soon as that happens,
       #abandon the whole loop and just give back the full payload without decompression
       if compressedSize<=0 or uncompressedSize<=0:
           f.seek(pos0)
           return f.read(size)
       #likewise, make sure that compressed size does not exceed the size of the file
       if f.tell()+compressedSize>pos0+size:
           f.seek(pos0)
           return f.read(size)

       #try to decompress
       if compressedSize!=uncompressedSize:
           try:    outStream.write(zlib.decompress(f.read(compressedSize)))
           except: outStream.write(f.read(compressedSize))
       else:
           #if compressed==uncompressed, one might be tempted to think that it is always non-zlib. It's not.
           magic=f.read(2)
           f.seek(-2,1)
           if magic=="\x78\xda":
               try:    outStream.write(zlib.decompress(f.read(compressedSize)))
               except: outStream.write(f.read(compressedSize))
           else:
               outStream.write(f.read(compressedSize))

   data=outStream.getvalue()
   outStream.close()
   return data

Note that the reason you get different errors all the time is because the script does not overwrite files. It will break at a file and leave a broken, empty file in your output folder. The next time the script is run it will see that the file already exists and skip it. For proper extraction you'll need to delete the output folder first. The reason I've set it to not overwrite files is because the archives are very redundant. Just think of the cascat archives which contain the content for all 10 or so vanilla maps. If overwriting was enabled, the script would copy the same content 10 times. So it's for performance reasons that I've turned that off. As a result, you will need to dump the patched files first, and then the unpatched ones. As the script works alphabetically, it does the patched vanilla maps and patched DLC first, then the unpatched DLC. The unpatched vanilla files then require a different tocRoot.

Edited by Frankelstner

Share this post


Link to post
Share on other sites

Deleted all previously extracted files. Tried dumping the patched files with your brand new zlib function but, after 127.247 files, last opened file: XP1_001.toc it gives the 'module' error:

Traceback (most recent call last):
 File "C:\Program Files (x86)\Origin Games\Battlefield 3\Update\New folder\dumper.py", line 405, in <module>
   main()
 File "C:\Program Files (x86)\Origin Games\Battlefield 3\Update\New folder\dumper.py", line 402, in main
   dump(fname,outputfolder)
 File "C:\Program Files (x86)\Origin Games\Battlefield 3\Update\New folder\dumper.py", line 300, in dump
   bundle=Bundle.Bundle(sb)
TypeError: 'module' object is not callable

Why always after 127.247 files, at the same point?

Thanks again mr Frank!

P.S. So, after the error, should I re-run the script?

P.P.S. After re-running it, it's even more clear the XP_001 is faulty. Which is the faulty file, the 3 KB toc or the over 700 MB sb? What do you think?

Edited by JackusCTB

Share this post


Link to post
Share on other sites

Deleted all previously extracted files. Tried dumping the patched files with your brand new zlib function but, after 127.247 files, last opened file: XP1_001.toc it gives the 'module' error:

...

Why always after 127.247 files, at the same point?

Thanks again mr Frank!

P.S. So, after the error, should I re-run the script?

P.P.S. After re-running it, it's even more clear the XP_001 is faulty. Which is the faulty file, the 3 KB toc or the over 700 MB sb? What do you think?

The DLC sbtoc are different from the vanilla sbtoc. I suspect that the script instantly fails at the DLC sbtoc, which require the Bundle.py file. About that file, it requires a capitalized "B" in the name so that might be the reason. Will look at other options too.

Basically when the script fails the safest approach would be to start from scratch with all files deleted though of course it's rather annoying to do this repeatedly.

And there is no point in distinguishing between toc (table of contents) and sb (superbundle). The toc contains the info to split the sb into individual bundles which then can be split into individual files. You can't have one without the other.

Share this post


Link to post
Share on other sites

The DLC sbtoc are different from the vanilla sbtoc. I suspect that the script instantly fails at the DLC sbtoc, which require the Bundle.py file. About that file, it requires a capitalized "B" in the name so that might be the reason. Will look at other options too.

Basically when the script fails the safest approach would be to start from scratch with all files deleted though of course it's rather annoying to do this repeatedly.

And there is no point in distinguishing between toc (table of contents) and sb (superbundle). The toc contains the info to split the sb into individual bundles which then can be split into individual files. You can't have one without the other.

I already had Bundle.py with the capitalized "B", so I think this shouldn't be the problem. Anyway, I deleted the XP_001 folders and now I'm repairing the install with Origin. 30 minutes left to download 2.1 GB.

Edited by JackusCTB

Share this post


Link to post
Share on other sites

I already had Bundle.py with the capitalized "B", so I think this shouldn't be the problem. Anyway, I deleted the XP_001 folders and now I'm repairing the install with Origin. 30 minutes left to download 2.1 GB.

The files are not the issue. I think there is some kind of mismatch between the Bundle.py and the dumper.py. Upload both of them to e.g. pastebin so I can take a look at them.

The files on the first page of this thread should work, though I'm curious to see what went wrong. Also, Python creates a .pyc whenever a script is imported from another script. The thing is, as soon as such a file is created, Python will only check for the existence of such a file and will not recreate the .pyc file. So it may be that the Bundle.py contains different instructions than the Bundle.pyc. Delete all .pycs to be sure; Python will recreate them as needed.

Edited by Frankelstner

Share this post


Link to post
Share on other sites

The files are not the issue. I think there is some kind of mismatch between the Bundle.py and the dumper.py. Upload both of them to e.g. pastebin so I can take a look at them.

The files on the first page of this thread should work, though I'm curious to see what went wrong. Also, Python creates a .pyc whenever a script is imported from another script. The thing is, as soon as such a file is created, Python will only check for the existence of such a file and will not recreate the .pyc file. So it may be that the Bundle.py contains different instructions than the Bundle.pyc. Delete all .pycs to be sure; Python will recreate them as needed.

These are the files I was about to upload, I read your mind! http://www.mediafire.com/?ier1a3r2st9bejv (Edited for better site hosting)

Edited by JackusCTB

Share this post


Link to post
Share on other sites

These are the files I was about to upload, I read your mind! http://www.mediafire.com/?ier1a3r2st9bejv (Edited for better site hosting)

What the heck? Bundle.py imports itself. Inception much?

Bundle.py:

import sys
import os
from struct import unpack, pack
from binascii import hexlify, unhexlify
import zlib
from cStringIO import StringIO
from collections import OrderedDict
import Bundle

Your Bundle.py contains the sbtoc.py script. You do not have an actual Bundle.py script.

Edited by Frankelstner

Share this post


Link to post
Share on other sites

What the heck? Bundle.py imports itself. Inception much?

Bundle.py:

import sys
import os
from struct import unpack, pack
from binascii import hexlify, unhexlify
import zlib
from cStringIO import StringIO
from collections import OrderedDict
import Bundle

You'll need to get rid of the last line. Or just use the script on the first page of the thread.

Err, I think you swapped sbtoc.py and Bundle.py.

My bad, I didn't even checked the files! I downloaded them from here http://www.bfeditor.org/forums/index.php?showtopic=15731&view=findpost&p=106679 because I was so confused that they didn't worked on my machine.

Thanks again mr. Frank... now I will try to see if now they work.

P.S. For the dumper script, which zlib function should I use? The old or the new one?

Edited by JackusCTB

Share this post


Link to post
Share on other sites
P.S. For the dumper script, which zlib function should I use? The old or the new one?

Try the new zlib function. It has worked pretty well so far.

And give feedback if everything works correctly. In that case I will update the OP with the new zlib function too.

Share this post


Link to post
Share on other sites

Try the new zlib function. It has worked pretty well so far.

And give feedback if everything works correctly. In that case I will update the OP with the new zlib function too.

Thanks very much, mr. Frank!

This time I successfully extracted all the data. First, I used the new zlib function and I extracted the update + DLC files in hexing\bf3 dump. Then, in the same folder, I extracted the original files, without any error!

Now I've got 213.346 files, for a total of 36.264.468.935 bytes (or 33.7 GB).

Attached you will find the dumper logs (vanilla + update) and the scripts I have used.

dumper logs

Battlefield 3 dumper

Share this post


Link to post
Share on other sites

Alright, thanks for testing. I've changed that zlib function in the dumper script. While the previous version did not cause issues for me at least, the new version should be generally more stable.

Share this post


Link to post
Share on other sites

You're welcome. Anyway, now to make sense, I should run the EBX File Converter and eventually the FrostBite 2 Audio Decoder or the files are already usable?

It depends on what you want to do. If you want to have a look at the ebx files converted to text, then run the ebx converter. If you want the audio, you do not need the ebx converter. I've copypasted some stuff from the ebx converter into the audio decoder, so it has its own ebx parser; in fact the audio decoder cannot comprehend the text files that you get when converting the ebx.

Share this post


Link to post
Share on other sites

It depends on what you want to do. If you want to have a look at the ebx files converted to text, then run the ebx converter. If you want the audio, you do not need the ebx converter. I've copypasted some stuff from the ebx converter into the audio decoder, so it has its own ebx parser; in fact the audio decoder cannot comprehend the text files that you get when converting the ebx.

Alright, thanks for the information.

Most of the textures can be extracted with daniuxxx's GUID reader, right? And for the meshes there isn't yet a way to decode them, right?

Thanks again!

Share this post


Link to post
Share on other sites

Alright, thanks for the information.

Most of the textures can be extracted with daniuxxx's GUID reader, right? And for the meshes there isn't yet a way to decode them, right?

Thanks again!

Correct and correct. I must admit I haven't tested his texture converter though, so I am not certain as to how many textures you will be able to convert.

Share this post


Link to post
Share on other sites

Mr. Frank, I've got a problem...

I converted the ebx files to txt and then converted the audio files with fb2decoder.py. It said that there weren't "some" chunks files. How can this be possible if dumper.py didn't gave me any error?

If you could maybe give a look at these logs, one from the ebx to txt converter and the other from fb2decoder.

Thanks!

Ebx to txt converter log

Frostbite 2 audio converter

Share this post


Link to post
Share on other sites

Mr. Frank, I've got a problem...

I converted the ebx files to txt and then converted the audio files with fb2decoder.py. It said that there weren't "some" chunks files. How can this be possible if dumper.py didn't gave me any error?

If you could maybe give a look at these logs, one from the ebx to txt converter and the other from fb2decoder.

Thanks!

Ebx to txt converter log

Frostbite 2 audio converter

http://www.bfeditor.org/forums/index.php?showtopic=15780&view=findpost&p=106613

Share this post


Link to post
Share on other sites

Alright, thanks. So those chunks I'm missing are just some localization files that I don't have due to my language, right?

Yes. I do not remember why I thought it was a good idea to print that message. It seemed important back then and I must have had a reason (hopefully), so I think it's wiser to leave that message as it is.

Share this post


Link to post
Share on other sites

With Dragon Age Inquisition recently coming out, should the tool be able to extract audio from the game? I've been fiddling around with the dumper.py script for a bit but I keep getting errors. Removing that .toc does allow me to continue but anytime I remove the mentioned file, it just keeps giving me a new error. I assume I could remove every single .toc file from the game folder but that just seems counter productive...

ambientencounters.toc

C:\dragonage_sounds\bundles\ebx\da3/lod_groups/levels/mesh/da3_model_prop_small.ebx

Traceback (most recent call last):

File "C:\bin\dumper.py", line 367, in

main()

File "C:\bin\dumper.py", line 364, in main

dump(fname,outputfolder)

File "C:\bin\dumper.py", line 163, in dump

handlePayload(entry,ebxPath+entry.elems["name"].content+".ebx")

File "C:\bin\dumper.py", line 356, in casHandlePayload

LZ77.decompressAndWriteFile(catEntry.filename,catEntry.offset,catEntry.size,originalSize,outPath)

WindowsError: [Error -529697949] Windows Error 0xE06D7363

Edited by bacon

Share this post


Link to post
Share on other sites

With Dragon Age Inquisition recently coming out, should the tool be able to extract audio from the game? I've been fiddling around with the dumper.py script for a bit but I keep getting errors. Removing that .toc does allow me to continue but anytime I remove the mentioned file, it just keeps giving me a new error. I assume I could remove every single .toc file from the game folder but that just seems counter productive...

I haven't ever worked with Python before, so I can't fix the script. However, I can point out something that you guys can add :

That 09 70 header that is in front of LZ77-compressed data? DA:I doesn't have that. We have 02 70 in our headers, and in these cases, the data is very clearly compressed with ZLIB. There's a 78 DA heading on the block.

I wrote a few crap tools to try to pull this stuff out of the CAS files in .NET. Decompression works (so long as you use DotNetZip and not the internal System.IO.Compression.DeflateStream class). ZLIB has apparently been part of Python's libraries for a long time, so the code should be able to be converted. My current guess is that you'll need to write a little more code to pull out the exact block that ZLIB needs to decrypt - if I'm reading the code right, this is done within the LZ77 library, so it'll need to be rewritten for use with ZLIB.

Edit : Just to illustrate, here's some stuff from that DA:I cas_01.cas file.

First 0x30 bytes :

FA CE 0F F0 15 D4 9E 9E

37 4B 3C F4 FC 06 9C 02

47 DC 8B 28 C3 D1 92 4D

C9 B5 00 00 00 00 00 00

00 01 00 00 02 70 AC F4

78 DA 3C 9C 0D 7C 8D F5

Same CAS header, same SHA1 hash. Block header starts at 0x20. 0x00010000 for the uncompressed length - yep, it's 64k after decompression. 0x0270 for zlib. 0xACF4 for compressed length - 44276 bytes.

Moving down 0xACF4 (44276) bytes, we see another header!

00 00 3D A7 02 70 08 C5

78 DA ED 5B

Exact same thing. I verified the compressed and uncompressed length here. This is probably the end of the asset in question.

Edited by wogoodes

Share this post


Link to post
Share on other sites

I dunno whether there are any demands for this... but here's something I cooked up a while ago.

http://dl.dropboxusercontent.com/s/l3o7kfjbuf1xn42/screenshot.png

It performs automatic DXT size/format detection.

It's written in VB6 (ugh!), so you might need to find the VB6 runtime.

There's also PHP equivalent of the code included.

https://www.dropbox.com/s/czec5utwftlljew/iTexDump.zip?dl=0

Edited by JohnMarkus

Share this post


Link to post
Share on other sites

 

Thanks very much, mr. Frank!

This time I successfully extracted all the data. First, I used the new zlib function and I extracted the update + DLC files in hexing\bf3 dump. Then, in the same folder, I extracted the original files, without any error!

Now I've got 213.346 files, for a total of 36.264.468.935 bytes (or 33.7 GB).

Attached you will find the dumper logs (vanilla + update) and the scripts I have used.

dumper logs

Battlefield 3 dumper

 

Hallo,

I've got a simmilar problem. Can you upload these files again? Because these files are not longer avaliable.

Thank You!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×