Log in

View Full Version : [WIP] Generic Xbox360 Compression Library



fairchild
09-16-2012, 07:37 PM
This is a Library written in (unmanaged) C/C++ that can be used in your .NET (C# example available) applications to decompress and compress the compressed data from the savegame.

Make sure to give credits if used in your application.

Update:

So, lzopro gave me a hard time as it's not ment to be used the way i wanted it. Took some time to fix this but here is a new version that should work as expected.

Example (C#):


[DllImport("compress.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern uint lzo_decompress([Out] byte[] compressed_buffer, uint compressed_size, [In] byte[] decompressed_buffer, uint decompressed_size, uint method);

[DllImport("compress.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern uint lzo_compress([Out] byte[] decompressed_buffer, uint decompressed_size, [In] byte[] compressed_buffer, uint compressed_size, uint method, uint compress_level);

void MyFunction()
{
UInt32 decompressed_size = <decompressed size>;
byte[] decompressed_buffer = new byte[decompressed_size];

uint compressed_size = <compressed size>;
uint result = lzo_decompress(compressed_buffer, compressed_size, decompressed_buffer, decompressed_size, COMP_LZOPRO1X);

// result now contains the real decompressed size and decompressed_buffer contains the decompressed data.
}


Available compression methods:


enum {
// decompressors
COMP_NONE = 0, // No compression
COMP_ZLIB, // RFC 1950
COMP_DEFLATE, // RFC 1951
COMP_LZO1, // LZO 1 Freeware
COMP_LZO1A, // LZO 1a Freeware
COMP_LZO1B, // LZO 1b (safe with overrun) Freeware
COMP_LZO1C, // LZO 1c (safe with overrun) Freeware
COMP_LZO1F, // LZO 1f (safe with overrun) Freeware
COMP_LZO1X, // LZO 1x (safe with overrun) Freeware
COMP_LZO1Y, // LZO 1y (safe with overrun) Freeware
COMP_LZO1Z, // LZO 1z (safe with overrun) Freeware
COMP_LZO2A, // LZO 2a (safe with overrun) Freeware
COMP_LZOPRO1X, // LZOPRO 1x (safe with overrun) Freeware
COMP_LZOPRO1Y, // LZOPRO 1y (safe with overrun) Freeware

// compressors
COMP_LZO1_COMPRESS, // LZO 1 Freeware
COMP_LZO1_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1A_COMPRESS, // LZO 1a Freeware
COMP_LZO1A_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1B_COMPRESS, // LZO 1b Freeware (Valid compression level: 1..9)
COMP_LZO1B_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1B_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1C_COMPRESS, // LZO 1c Freeware (Valid compression level: 1..9)
COMP_LZO1C_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1C_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1F_COMPRESS, // LZO 1f Freeware
COMP_LZO1F_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1X_COMPRESS, // LZO 1x Freeware
COMP_LZO1X_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1Y_COMPRESS, // LZO 1y Freeware
COMP_LZO1Y_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1Z_COMPRESS, // LZO 1z Freeware
COMP_LZO1Z_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO2A_COMPRESS, // LZO 2a Freeware
COMP_LZO2A_999_COMPRESS, // even better compression ratio at the cost of more memory and time
COMP_LZOPRO1X_COMPRESS, // LZOPRO 1x (Valid compression level: 1..10)
COMP_LZOPRO1Y_COMPRESS, // LZOPRO 1y (Valid compression level: 1..10)
};

CloudStrife7x
09-16-2012, 07:47 PM
awesome work on this fair :076::029:

thegutta
09-16-2012, 09:39 PM
Excellent work Fairchild :)

Serious_M
09-17-2012, 07:24 PM
very nice work as always fairchild

kill_seth
09-19-2012, 04:55 AM
Can't seem to get it working in VB.Net



<DllImport("compress.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Shared Function lzo_decompress(<Out> compressed_buffer As Byte(), compressed_size As UInteger, <[In]> decompressed_buffer As Byte(), decompressed_size As UInteger, type As UInteger) As UInteger
End Function

Private Function Decompress(compressedData As Byte()) As Byte()
Dim decompressedSize As UInt32 = dSize
Dim decompressedData As Byte() = New Byte(decompressedSize) {}
Dim compressedSize As UInt32 = cSize
lzo_decompress(compressedData, compressedSize, decompressedData, decompressedSize, 0)
Return decompressedData
End Function


When I go to call the function I get: Unable to load DLL 'compress.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

I placed the dll in the Application.StartUpPath and in Windows\System32

PUR3 RAF3X
09-19-2012, 09:41 AM
Yes the same prob here. I have try use another ways but nothing work with this dll.




Imports PackageIO
Imports System.Text
Imports System
Imports System.Runtime.InteropServices


Public Declare Function lzo_decompress Lib "C:\Users\****\Documents\Visual Studio 2010\Projects\WindowsApplication5\WindowsApplicati on5\obj\x86\Debug\compress.dll" (ByVal Compressed_Buffer() As Byte, ByVal Compressed_Buffersize As UInteger, ByVal decompressed_buffer() As Byte, ByVal decompressed_size As UInteger, ByVal type As UInteger) As Integer



<DllImport("compress.dll", CharSet:=CharSet.Auto, CallingConvention:=CallingConvention.Cdecl)> _
Public Shared Function lzo_decompress(<Out()> compressed_buffer As Byte(), compressed_size As UInteger, <[In]()> decompressed_buffer As Byte(), decompressed_size As UInteger, type As UInteger) As UInteger
End Function






Dim FilePath As String

Dim Compressed_Buffer() As Byte
Dim decompressed_size As UInteger
Dim compressed_size As UInteger

Private Sub Decompressdata()

Dim reader As New PackageIO.Reader(FilePath, Endian.Big)
reader.Position = 28
decompressed_size = reader.ReadInt32()
Dim decompressed_buffer As Byte() = New Byte((decompressed_size)) {}
Compressed_Buffer = reader.ReadBytes(reader.Length, Endian.Little)
compressed_size = Compressed_Buffer.Length
decompressed_buffer = {(lzo_decompress(Compressed_Buffer, compressed_size, decompressed_buffer, decompressed_size, 0))}
System.IO.File.WriteAllBytes(Application.StartupPa th + "/SaveGame.dat", decompressed_buffer)

End Sub

Jappi88
09-19-2012, 11:20 AM
U guys can try whatever u want...it wont work...its not the way u coded it...fairchild forget to inlcude the maindll.

PUR3 RAF3X
09-21-2012, 12:02 PM
Whats happen with this dll?

Jappi88
09-21-2012, 02:44 PM
Whats happen with this dll?

Be Patient...Fairchild will fix it when he gets the time.

Also dont get your hopes up , u most probebly wont be able to desompress the save using the compress.dll(LzoPro)

fairchild
09-26-2012, 10:41 AM
Compression library has been updated now, hope everything works smooth now.

Note the new argument (uint compress_level) on the compress function.
Valid compress levels are (LZO Freeware: 1-9, LZO PRO: 1-10)

mojobojo
09-26-2012, 11:51 AM
The readme has a typo, the VB code example is copied twice. Also wouldn't we need an h file for the lib file to include the lib file? Anways, great job.

kill_seth
09-26-2012, 11:52 AM
Just tested out the compression, and it seems either I'm doing something wrong or it's still not working properly.



<DllImport("compress.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Shared Function lzo_compress(<Out> decompressed_buffer As Byte(), decompressed_size As UInteger, <[In]> compressed_buffer As Byte(), compressed_size As UInteger, type As UInteger, compression_level As UInteger) As UInteger
End Function

Private Function Compress(decompressedData As Byte()) As Byte()
Dim compressedData As Byte() = New Byte(compressedSize - 1) {}
lzo_compress(decompressedData, decompressedSize, compressedData, compressedSize, 12, 1)
Return compressedData
End Function


It's just returning a null byte array. I tried doing different compression levels, and also tried 13 but got the same result.

fairchild
09-26-2012, 01:00 PM
The library seems to be unstable, need to look into more error handling when sending signals from unmanaged code to managed, but the library works in my example C# tool i made really fast. Perhaps the sourcecode below helps:



using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using System.IO;

namespace example
{
public partial class Form1 : Form
{
enum CompressionMethods {
// decompressors
COMP_NONE = 0, // No compression
COMP_ZLIB, // RFC 1950
COMP_DEFLATE, // RFC 1951
COMP_LZO1, // LZO 1 Freeware
COMP_LZO1A, // LZO 1a Freeware
COMP_LZO1B, // LZO 1b (safe with overrun) Freeware
COMP_LZO1C, // LZO 1c (safe with overrun) Freeware
COMP_LZO1F, // LZO 1f (safe with overrun) Freeware
COMP_LZO1X, // LZO 1x (safe with overrun) Freeware
COMP_LZO1Y, // LZO 1y (safe with overrun) Freeware
COMP_LZO1Z, // LZO 1z (safe with overrun) Freeware
COMP_LZO2A, // LZO 2a (safe with overrun) Freeware
COMP_LZOPRO1X, // LZOPRO 1x (safe with overrun) Freeware
COMP_LZOPRO1Y, // LZOPRO 1y (safe with overrun) Freeware

// compressors
COMP_LZO1_COMPRESS, // LZO 1 Freeware
COMP_LZO1_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1A_COMPRESS, // LZO 1a Freeware
COMP_LZO1A_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1B_COMPRESS, // LZO 1b Freeware (Valid compression level: 1..9)
COMP_LZO1B_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1B_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1C_COMPRESS, // LZO 1c Freeware (Valid compression level: 1..9)
COMP_LZO1C_99_COMPRESS, // better compression ratio at the cost of more memory and time
COMP_LZO1C_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1F_COMPRESS, // LZO 1f Freeware
COMP_LZO1F_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1X_COMPRESS, // LZO 1x Freeware
COMP_LZO1X_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1Y_COMPRESS, // LZO 1y Freeware
COMP_LZO1Y_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO1Z_COMPRESS, // LZO 1z Freeware
COMP_LZO1Z_999_COMPRESS,// even better compression ratio at the cost of more memory and time
COMP_LZO2A_COMPRESS, // LZO 2a Freeware
COMP_LZO2A_999_COMPRESS, // even better compression ratio at the cost of more memory and time
COMP_LZOPRO1X_COMPRESS, // LZOPRO 1x (Valid compression level: 1..10)
COMP_LZOPRO1Y_COMPRESS, // LZOPRO 1y (Valid compression level: 1..10)
}

public byte[] ReadAllBytes(string fileName, int offset)
{
byte[] buffer = null;
using (FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
//Console.Out.WriteLine("fs.Length: " + fs.Length + "\n");
//Console.Out.WriteLine("offset: " + offset + "\n");
buffer = new byte[(fs.Length - offset)];
fs.Seek(offset, SeekOrigin.Begin);
fs.Read(buffer, 0, (int)(fs.Length - offset));
}
return buffer;
}

public Form1()
{
InitializeComponent();
}

[DllImport("compress.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern uint lzo_decompress([Out] byte[] compressed_buffer, uint compressed_size, [In] byte[] decompressed_buffer, uint decompressed_size, uint method);

[DllImport("compress.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern uint lzo_compress([Out] byte[] decompressed_buffer, uint decompressed_size, [In] byte[] compressed_buffer, uint compressed_size, uint method, uint compress_level);

private void button1_Click(object sender, EventArgs e)
{
string Pfad = string.Empty;

OpenFileDialog openFileDialog1 = new OpenFileDialog();
openFileDialog1.Filter = "Save (*.sav)|*.sav|All files (*.*)|*.*";
if (openFileDialog1.ShowDialog() == DialogResult.OK)
Pfad = openFileDialog1.FileName;

UInt32 decompressed_size = 24875; // Static for now on my testfile
byte[] decompressed_buffer = new byte[decompressed_size];

byte[] compressed_buffer = ReadAllBytes(Pfad, 0x18);

uint compressed_size = (uint)compressed_buffer.Length;
uint result = lzo_decompress(compressed_buffer, compressed_size, decompressed_buffer, decompressed_size, (uint)CompressionMethods.COMP_LZOPRO1X);

richTextBox1.Text += "\nDecompress result: " + result + "\n";
for (int i = 0; i < 128; i++)
richTextBox1.Text += decompressed_buffer[i].ToString("X2");


decompressed_size = result;
result = lzo_compress(decompressed_buffer, decompressed_size, compressed_buffer, compressed_size, (uint)CompressionMethods.COMP_LZOPRO1X_COMPRESS, 10);

richTextBox1.Text += "\nCompress result: " + result + "\n";
for (int i = 0; i < 128; i++)
richTextBox1.Text += compressed_buffer[i].ToString("X2");


/*
for (int i = 0; i < result; i++)
richTextBox1.Text += decompressed_buffer[i].ToString("X2");
*/

}
}
}

PUR3 RAF3X
09-26-2012, 01:36 PM
I try again thanks hope we can De/Compress Borderland 2 ;)

PUR3 RAF3X
09-26-2012, 03:37 PM
No idea why but the decompressedData return allways no data? Any idea?

fairchild
09-26-2012, 05:00 PM
No idea why but the decompressedData return allways no data? Any idea?

Don't know, perhaps it's correct but why [] on the In and not the Out and should the [] be there for VB?


<Out> decompressed_buffer As Byte(), decompressed_size As UInteger, <[In]> compressed_buffer

Jappi88
09-26-2012, 05:52 PM
this code is for vb to call the unmanaged decompress function >>:


<DllImport("compress.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Shared Function lzo_compress(<Out()> ByVal decompressed_buffer As Byte(), ByVal decompressed_size As Integer, <[In]()> ByVal compressed_buffer As Byte(), ByVal compressed_size As Integer, ByVal type As Integer) As Integer
End Function


if the decompress Returns nulled or invalid data means the file ur decompressing does not use that type.
so to check if ur decompressing whent like it suposed to be, could be done like this :



'declare a varaible as integer to use for return result from the decompressor
Dim Result as Integer = lzo_decompress(buffer, buffer.Length, Decompressed, Length, 8)

if Result > -1 then msgbox("Succesfull Decompression") else msgbox("Failed decompressing file!")


if the decompressing when succesfully it will return the decompressed length else it will return a negative value. (-1)

PUR3 RAF3X
09-26-2012, 07:37 PM
lmao my vault I have do a noob step this is why I have trouble but guys it's work and I can Decompress & Compress the Borderland 2 save. When kill_seph not update his Tool then I release my one and big big thanks to fairchild. All credits to him ;)

Here the translated code from fairchild to vb:


Imports System.Collections.Generic
Imports System.ComponentModel
Imports System.Data
Imports System.Drawing
Imports System.Linq
Imports System.Text
Imports System.Windows.Forms
Imports System.Runtime.InteropServices
Imports System.IO

Namespace example
Public Partial Class Form1
Inherits Form
Private Enum CompressionMethods
' decompressors
COMP_NONE = 0
' No compression
COMP_ZLIB
' RFC 1950
COMP_DEFLATE
' RFC 1951
COMP_LZO1
' LZO 1 Freeware
COMP_LZO1A
' LZO 1a Freeware
COMP_LZO1B
' LZO 1b (safe with overrun) Freeware
COMP_LZO1C
' LZO 1c (safe with overrun) Freeware
COMP_LZO1F
' LZO 1f (safe with overrun) Freeware
COMP_LZO1X
' LZO 1x (safe with overrun) Freeware
COMP_LZO1Y
' LZO 1y (safe with overrun) Freeware
COMP_LZO1Z
' LZO 1z (safe with overrun) Freeware
COMP_LZO2A
' LZO 2a (safe with overrun) Freeware
COMP_LZOPRO1X
' LZOPRO 1x (safe with overrun) Freeware
COMP_LZOPRO1Y
' LZOPRO 1y (safe with overrun) Freeware
' compressors
COMP_LZO1_COMPRESS
' LZO 1 Freeware
COMP_LZO1_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1A_COMPRESS
' LZO 1a Freeware
COMP_LZO1A_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1B_COMPRESS
' LZO 1b Freeware (Valid compression level: 1..9)
COMP_LZO1B_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1B_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1C_COMPRESS
' LZO 1c Freeware (Valid compression level: 1..9)
COMP_LZO1C_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1C_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1F_COMPRESS
' LZO 1f Freeware
COMP_LZO1F_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1X_COMPRESS
' LZO 1x Freeware
COMP_LZO1X_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1Y_COMPRESS
' LZO 1y Freeware
COMP_LZO1Y_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1Z_COMPRESS
' LZO 1z Freeware
COMP_LZO1Z_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO2A_COMPRESS
' LZO 2a Freeware
COMP_LZO2A_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZOPRO1X_COMPRESS
' LZOPRO 1x (Valid compression level: 1..10)
COMP_LZOPRO1Y_COMPRESS
' LZOPRO 1y (Valid compression level: 1..10)
End Enum

Public Function ReadAllBytes(fileName As String, offset As Integer) As Byte()
Dim buffer As Byte() = Nothing
Using fs As New FileStream(fileName, FileMode.Open, FileAccess.Read)
'Console.Out.WriteLine("fs.Length: " + fs.Length + "\n");
'Console.Out.WriteLine("offset: " + offset + "\n");
buffer = New Byte((fs.Length - offset) - 1) {}
fs.Seek(offset, SeekOrigin.Begin)
fs.Read(buffer, 0, CInt(fs.Length - offset))
End Using
Return buffer
End Function

Public Sub New()
InitializeComponent()
End Sub

<DllImport("compress.dll", CallingConvention := CallingConvention.Cdecl)> _
Public Shared Function lzo_decompress(<Out> compressed_buffer As Byte(), compressed_size As UInteger, <[In]> decompressed_buffer As Byte(), decompressed_size As UInteger, method As UInteger) As UInteger
End Function

<DllImport("compress.dll", CallingConvention := CallingConvention.Cdecl)> _
Public Shared Function lzo_compress(<Out> decompressed_buffer As Byte(), decompressed_size As UInteger, <[In]> compressed_buffer As Byte(), compressed_size As UInteger, method As UInteger, compress_level As UInteger) As UInteger
End Function

Private Sub button1_Click(sender As Object, e As EventArgs)
Dim Pfad As String = String.Empty

Dim openFileDialog1 As New OpenFileDialog()
openFileDialog1.Filter = "Save (*.sav)|*.sav|All files (*.*)|*.*"
If openFileDialog1.ShowDialog() = DialogResult.OK Then
Pfad = openFileDialog1.FileName
End If

Dim decompressed_size As UInt32 = 24875
' Static for now on my testfile
Dim decompressed_buffer As Byte() = New Byte(decompressed_size - 1) {}

Dim compressed_buffer As Byte() = ReadAllBytes(Pfad, &H18)

Dim compressed_size As UInteger = CUInt(compressed_buffer.Length)
Dim result As UInteger = lzo_decompress(compressed_buffer, compressed_size, decompressed_buffer, decompressed_size, CUInt(CompressionMethods.COMP_LZOPRO1X))

richTextBox1.Text += vbLf & "Decompress result: " & result & vbLf
For i As Integer = 0 To 127
richTextBox1.Text += decompressed_buffer(i).ToString("X2")
Next


decompressed_size = result
result = lzo_compress(decompressed_buffer, decompressed_size, compressed_buffer, compressed_size, CUInt(CompressionMethods.COMP_LZOPRO1X_COMPRESS), 10)

richTextBox1.Text += vbLf & "Compress result: " & result & vbLf
For i As Integer = 0 To 127
richTextBox1.Text += compressed_buffer(i).ToString("X2")
Next


'
' for (int i = 0; i < result; i++)
' richTextBox1.Text += decompressed_buffer[i].ToString("X2");
'


End Sub
End Class
End Namespace

kill_seth
09-26-2012, 08:08 PM
Did some more testing, and it seems the library isn't working correctly with the compression.



<DllImport("compress.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Shared Function lzo_compress(<Out> decompressed_buffer As Byte(), decompressed_size As UInteger, <[In]> compressed_buffer As Byte(), compressed_size As UInteger, type As UInteger, compression_level As UInteger) As UInteger
End Function

If OFD.ShowDialog = Windows.Forms.DialogResult.OK Then
gReader = New GPDPackage.GPD(OFD.FileName)
iReader = New IOPackage.RWStream(gReader.ExtractSaveData(GPDPack age.GPD.SaveData.TitleSpecific1))
SHA1 = iReader.ReadBytes(&H14)
decompressedSize = iReader.ReadUInt32()
Dim decompressedData As Byte() = New Byte(decompressedSize - 1) {}
Dim compressedData As Byte() = iReader.ReadBytes(iReader.Length - iReader.Position)
compressedSize = compressedData.Length
File.WriteAllBytes("Profile(compressed)", compressedData)
Dim result = lzo_decompress(compressedData, compressedSize, decompressedData, decompressedSize, 12)
Dim result2 = lzo_compress(decompressedData, decompressedSize, compressedData, compressedSize, 12, 10)
File.WriteAllBytes("Profile(decompressed)", decompressedData)
File.WriteAllBytes("Profile(recompressed)", compressedData)
End If


Doing that both the Profile(compressed) and Profile(recompressed) files are exactly the same. So at first, I thought I finally got it to work and decided to try something else. I edited the decompressed data and wrote a 1 to the first uint32 entry, then tried to recompress it. Turns out, both the files were exactly the same again. I even tried creating a recompressedData and recompressedSize, but then I just ended up with a null bytes once again. Thanks for all your hard work, and I hope we can figure this out!

PUR3 RAF3X
09-26-2012, 09:53 PM
Yes this is hard and make me sick.

PUR3 RAF3X
09-27-2012, 09:43 AM
@kill the vault why it's return 0 is when you set CompressionMethods to 0 or none then it's return all bytes with "0" use for decompress CompressionMethods.COMP_LZOPRO1X or try around with another Enums. But I have the best result with Decompress this data using my own LZO 2.06 with custom setting.

eyebiz
09-27-2012, 11:16 AM
I must be missing something. How could you possibly know the size "decompressed_size" of the decompressed_buffer before decompression?

Edit: When working with a pre-compressed file, aka the BL2 save.

Jappi88
09-27-2012, 11:19 AM
I must be missing something. How could you possibly know the size "decompressed_size" of the decompressed_buffer before decompression?

Because in some compressions the decompressed size is discripted inside the data.
Lzo works that way.

@Pur3
there is nothing wrong with compress.dll(lzopro) works like it should... It jyst doesnt have the suported compression type for borderlands 2.

PUR3 RAF3X
09-28-2012, 10:04 AM
Because in some compressions the decompressed size is discripted inside the data.
Lzo works that way.

@Pur3
there is nothing wrong with compress.dll(lzopro) works like it should... It jyst doesnt have the suported compression type for borderlands 2.

Ahh ok. Yes I see this the dll do the job and I have include this to my Tool so the guys can use it and try self around. The Tool support Packzip/offzip and this lib ;) work in progress are xextool I now many will love this.

PUR3 RAF3X
09-28-2012, 03:28 PM
I must be missing something. How could you possibly know the size "decompressed_size" of the decompressed_buffer before decompression?

Edit: When working with a pre-compressed file, aka the BL2 save.





LZO implements a number of algorithms with the following features:

Decompression is simple and *very* fast.
Requires no memory for decompression.
Compression is pretty fast.
Requires 64 kB of memory for compression.
Allows you to dial up extra compression at a speed cost in the compressor. The speed of the decompressor is not reduced.
Includes compression levels for generating pre-compressed data which achieve a quite competitive compression ratio.
There is also a compression level which needs only 8 kB for compression.
Algorithm is thread safe.
Algorithm is lossless.

LZO supports overlapping compression and in-place decompression.


Maybe this help you.

PUR3 RAF3X
09-30-2012, 01:10 PM
For Borderlands you can do the Decompress like this sample: But have any try compress Decompress big files over 100mb? This is my first time I work with Compress so any help welcome. I have try fill the buffer with 8kb and copy the compressed 6kb array to the 8kb buffer and try decompress this. The result are the same 6kb or 8kb lzo skip all "0" after the 6kb and return the same decompressed bytes.

EDIT:I have my answer and I work on my own Module doing this with files over 8GB ;) Using custom header for files over 512MB. You guys can downloade the source later.


Private Sub DecompressButton_Click_1(sender As System.Object, e As System.EventArgs) Handles DecompressButton.Click
try
If Not DecompressInputPath = Nothing Then

Writer = New BinaryWriter(New FileStream(Me.DecompressSavePath, FileMode.Create, FileAccess.Write, FileShare.Write))
Dim mblock As Integer = 8192
Dim bsize As Integer = 8192 * 8 '/64kb 8kb * 8 = 64kb
Dim buffer As Byte() = New Byte(bsize - 1) {}
Dim result As UInteger = 0
Dim compressed_buffer As Byte() = ReadBufferBytes(DecompressInputPath, 24,mblock)
result = lzo_decompress(compressed_buffer, compressed_buffer.Length, buffer, buffer.Length, _
CUInt(CompressionMethodsdecompress(ComboBox1.Text, True)))

If Not result = UInteger.MaxValue Then
Writer.Seek(&H0, SeekOrigin.Begin)
Writer.Write(buffer, 0, CInt(result))

Else
MsgBox("Decompress Failed")
End If

End If
Writer.Flush()
Writer.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
End Sub

JizzaBeez
10-07-2012, 09:56 PM
Farichild: how do I know the "compressed_size" when using lzo_compress ? Wouldn't the size be determined by input, method and level?

eyebiz
10-08-2012, 08:18 AM
Farichild: how do I know the "compressed_size" when using lzo_compress ? Wouldn't the size be determined by input, method and level?

I used the same as "decompressed_size" and then just resized the array to the "result" value after compression.

PUR3 RAF3X
10-08-2012, 08:45 AM
Yes the compress methode have different output size. I have no idea if there any another way doing this but my way I doing this is create a buffer with bytes and use UINT result for the info "Compress size". Thats all you can do the same again with the right info from result or you copy the compress data array from the buffer you have created for the info or write only the size from byte array there more then 1 way doing this and Im sure you found your right way what you need ;)

Im not the dude hide all my source so here my "WIP" Module for Compress big files with custom header this store each compress buffer size how many blocks and the memblock size. Only when the file size more then 256Mb.
I have only probs with the damn Progress inside the R_textbox for compress the data. reade byte work well.


Imports System.Collections.Generic
Imports System.ComponentModel
Imports System.Data
Imports System.Linq
Imports System.Text
Imports System.Windows.Forms
Imports System.Runtime.InteropServices
Imports System.IO



Module Compresslib

<DllImport("Plugins\compress.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Function lzo_decompress(<Out()> compressed_buffer As Byte(), compressed_size As UInteger, <[In]()> decompressed_buffer As Byte(), decompressed_size As UInteger, method As UInteger) As UInteger
End Function


<DllImport("Plugins\compress.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Function lzo_compress(<Out()> decompressed_buffer As Byte(), decompressed_size As UInteger, <[In]()> compressed_buffer As Byte(), compressed_size As UInteger, method As UInteger, compress_level As UInteger) As UInteger
End Function



Private bw As BinaryWriter
Private strb As New System.Text.StringBuilder

Private multiblocks As Boolean = False



Dim InputPath As String
Dim OutputPath As String

Dim endfile As Boolean = False

Dim Mem As Integer = 268435456
Dim calculatedblocks As Integer
Dim Lastblocksize As UInteger = 0
Dim Headersize As Integer = 0
Dim blockstartpos As String = &H0



Private Enum CompressionMethods
' decompressors
COMP_NONE = 0
' No compression
COMP_ZLIB
' RFC 1950
COMP_DEFLATE
' RFC 1951
COMP_LZO1
' LZO 1 Freeware
COMP_LZO1A
' LZO 1a Freeware
COMP_LZO1B
' LZO 1b (safe with overrun) Freeware
COMP_LZO1C
' LZO 1c (safe with overrun) Freeware
COMP_LZO1F
' LZO 1f (safe with overrun) Freeware
COMP_LZO1X
' LZO 1x (safe with overrun) Freeware
COMP_LZO1Y
' LZO 1y (safe with overrun) Freeware
COMP_LZO1Z
' LZO 1z (safe with overrun) Freeware
COMP_LZO2A
' LZO 2a (safe with overrun) Freeware
COMP_LZOPRO1X
' LZOPRO 1x (safe with overrun) Freeware
COMP_LZOPRO1Y
' LZOPRO 1y (safe with overrun) Freeware
' compressors
COMP_LZO1_COMPRESS
' LZO 1 Freeware
COMP_LZO1_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1A_COMPRESS
' LZO 1a Freeware
COMP_LZO1A_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1B_COMPRESS
' LZO 1b Freeware (Valid compression level: 1..9)
COMP_LZO1B_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1B_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1C_COMPRESS
' LZO 1c Freeware (Valid compression level: 1..9)
COMP_LZO1C_99_COMPRESS
' better compression ratio at the cost of more memory and time
COMP_LZO1C_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1F_COMPRESS
' LZO 1f Freeware
COMP_LZO1F_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1X_COMPRESS
' LZO 1x Freeware
COMP_LZO1X_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1Y_COMPRESS
' LZO 1y Freeware
COMP_LZO1Y_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO1Z_COMPRESS
' LZO 1z Freeware
COMP_LZO1Z_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZO2A_COMPRESS
' LZO 2a Freeware
COMP_LZO2A_999_COMPRESS
' even better compression ratio at the cost of more memory and time
COMP_LZOPRO1X_COMPRESS
' LZOPRO 1x (Valid compression level: 1..10)
COMP_LZOPRO1Y_COMPRESS
' LZOPRO 1y (Valid compression level: 1..10)
End Enum




Sub Compressfile(CompressInputPath As String, CompressSavePath As String)
InputPath = CompressInputPath
OutputPath = CompressSavePath
Calculatefilecompression()
StartCompression()
'Verifyheader()

End Sub

#Region "Compress Methoden"

Private Function CompressionMethodsdecompress(p1 As Object, ByVal ReturnInt As Boolean) As Object
If ReturnInt Then
Select Case p1
Case "COMP_LZOPRO1X"
Return CompressionMethods.COMP_LZOPRO1X
Case "COMP_LZOPRO1Y"
Return CompressionMethods.COMP_LZOPRO1Y
Case "COMP_LZO2A"
Return CompressionMethods.COMP_LZO2A
Case "COMP_LZO1Z"
Return CompressionMethods.COMP_LZO1Z
Case "COMP_LZO1Y"
Return CompressionMethods.COMP_LZO1Y
Case "COMP_LZO1X"
Return CompressionMethods.COMP_LZO1X
Case "COMP_LZO1F"
Return CompressionMethods.COMP_LZO1F
Case "COMP_LZO1C"
Return CompressionMethods.COMP_LZO1C
Case "COMP_LZO1B"
Return CompressionMethods.COMP_LZO1B
Case "COMP_LZO1A"
Return CompressionMethods.COMP_LZO1A
Case "COMP_LZO1"
Return CompressionMethods.COMP_LZO1
Case "COMP_DEFLATE"
Return CompressionMethods.COMP_DEFLATE
Case "COMP_ZLIB"
Return CompressionMethods.COMP_ZLIB
Case "COMP_NONE"
Return CompressionMethods.COMP_NONE
Case Else
xb360Generic.RichTextBox1.AppendText("Wrong CompressionMethods" & xb360Generic.ComboBox1.Text.ToString)
Return False
End Select

End If
Return Nothing

End Function

Private Function CompressionMethodscompress(p1 As Object, ByVal ReturnInt As Boolean) As Object 'Compress
If ReturnInt Then
Select Case p1
Case "COMP_LZO1_COMPRESS"
Return CompressionMethods.COMP_LZO1_COMPRESS
Case "COMP_LZO1_99_COMPRESS"
Return CompressionMethods.COMP_LZO1_99_COMPRESS
Case "COMP_LZO1A_COMPRESS"
Return CompressionMethods.COMP_LZO1A_COMPRESS
Case "COMP_LZO1A_99_COMPRESS"
Return CompressionMethods.COMP_LZO1A_99_COMPRESS
Case "COMP_LZO1B_COMPRESS"
Return CompressionMethods.COMP_LZO1B_COMPRESS
Case "COMP_LZO1B_99_COMPRESS"
Return CompressionMethods.COMP_LZO1B_99_COMPRESS
Case "COMP_LZO1B_999_COMPRESS"
Return CompressionMethods.COMP_LZO1B_999_COMPRESS
Case "COMP_LZO1C_COMPRESS"
Return CompressionMethods.COMP_LZO1C_COMPRESS
Case "COMP_LZO1C_99_COMPRESS"
Return CompressionMethods.COMP_LZO1C_99_COMPRESS
Case "COMP_LZO1C_999_COMPRESS"
Return CompressionMethods.COMP_LZO1C_999_COMPRESS
Case "COMP_LZO1F_COMPRESS"
Return CompressionMethods.COMP_LZO1F_COMPRESS
Case "COMP_LZO1F_999_COMPRESS"
Return CompressionMethods.COMP_LZO1F_999_COMPRESS
Case "COMP_LZO1X_COMPRESS"
Return CompressionMethods.COMP_LZO1X_COMPRESS
Case "COMP_LZO1X_999_COMPRESS"
Return CompressionMethods.COMP_LZO1X_999_COMPRESS
Case "COMP_LZO1Y_COMPRESS"
Return CompressionMethods.COMP_LZO1Y_COMPRESS
Case "COMP_LZO1Y_999_COMPRESS"
Return CompressionMethods.COMP_LZO1Y_999_COMPRESS
Case "COMP_LZO1Z_COMPRESS"
Return CompressionMethods.COMP_LZO1Z_COMPRESS
Case "COMP_LZO1Z_999_COMPRESS"
Return CompressionMethods.COMP_LZO1Z_999_COMPRESS
Case "COMP_LZO2A_COMPRESS"
Return CompressionMethods.COMP_LZO2A_COMPRESS
Case "COMP_LZO2A_999_COMPRESS"
Return CompressionMethods.COMP_LZO2A_999_COMPRESS
Case "COMP_LZOPRO1X_COMPRESS"
Return CompressionMethods.COMP_LZOPRO1X_COMPRESS
Case "COMP_LZOPRO1Y_COMPRESS"
Return CompressionMethods.COMP_LZOPRO1Y_COMPRESS

Case Else
xb360Generic.RichTextBox1.AppendText("Wrong CompressionMethods" & xb360Generic.ComboBox1.Text.ToString)
Return False
End Select

End If
Return Nothing

End Function

#End Region




Private Sub Calculatefilecompression()
'/256MB
Dim L As String = FileSystem.FileLen(InputPath) '/File

strb.AppendLine()
strb.Append("Scan File......." & vbNewLine)
strb.Append(InputPath & vbNewLine)
strb.AppendLine()
strb.Append("Scan File.....done!" & vbNewLine)
strb.Append("Input:---> " & L & "(bytes)" & vbNewLine)
logMessage(strb.ToString)
strb.Clear()
If L < Mem Then
'/1Block
multiblocks = False
calculatedblocks = 1
Lastblocksize = L
Headersize = 0
blockstartpos = "&H0"

strb.Append("Calculated block:---> " & "(" & calculatedblocks & ")" & vbNewLine)
strb.Append("Use block size:---> " & "536870912(bytes)" & vbNewLine)
strb.Append("Header size:---> " & Headersize & "(bytes)" & vbNewLine)
strb.Append("1Block offset:---> " & blockstartpos & vbNewLine)
logMessage(strb.ToString)
strb.Clear()
Else
'/Multi Blocks calculate
multiblocks = True
calculatedblocks = Math.Floor(CStr(L) / CDec(Mem))
'/verhinderung falscher block berechnung
Dim c1 As String = (CStr(calculatedblocks * 256) * CDec(1024) * CDec(1024))
Lastblocksize = (CStr(L) - CDec(c1)) '/bytes last block for compress
If Not Lastblocksize = 0 Then calculatedblocks += 1
Headersize = CStr(calculatedblocks * 4) + (CDec(6)) '/Header berechnen 1block hat 4 bytes für compressed size / 2bytes für blockzahl / 2bytes für block size in MB
blockstartpos = "&H" & CStr(Hex$(blockstartpos) + Hex$(Headersize))
strb.Append("Calculated block:---> " & "(" & calculatedblocks & ")" & vbNewLine)
strb.Append("Use block size:---> " & "536870912(bytes)" & vbNewLine)
strb.Append("Header size:---> " & Headersize & "(bytes)" & vbNewLine)
strb.Append("Block @ offset:---> " & blockstartpos & vbNewLine)
logMessage(strb.ToString)
strb.Clear()
End If


End Sub

Dim positionheader As String = "&H0"
Dim sizeblock As UInteger = 0


Private _Maximum As Integer = 100
Private Percent As Integer = 0
Dim bytesRead As UInteger


Public Function ReadBufferBytes(Input As String, offset As String, sizeblock As UInteger) As Byte()
Dim buffer As Byte() = Nothing
Using fs As New FileStream(Input, FileMode.Open, FileAccess.Read, FileShare.Read)
strb.Append("Reade buffer bytes:---> " & sizeblock & "(bytes)" & vbNewLine)
logMessage(strb.ToString)
strb.Clear()


xb360Generic.ProgressBar1.Maximum = _Maximum
buffer = New Byte((sizeblock - 1)) {}
fs.Seek(offset, SeekOrigin.Begin)
strb.Append("Reade buffer bytes:---> " & Percent & "%" & vbNewLine)
logMessage(strb.ToString)
strb.Clear()

For i As Long = 0 To sizeblock
bytesRead = fs.Read(buffer, 0, CInt(sizeblock))
'/ update bei 1 Kb oder ende sizeblock
If i Mod 1024 = 0 Or i Mod sizeblock = 0 Then
fortschrittrb(i, sizeblock)

End If
Next


End Using
Return buffer
End Function

Private Function fortschrittrb(currentRead As UInteger, maxRead As UInteger)
Dim current As Integer
current = CDec(currentRead / maxRead * 100)
xb360Generic.RichTextBox2.Undo()'/Lazy clear last line from rtb
strb.Append("Progress:---> " & current & "%" & vbNewLine)
logMessage(strb.ToString)
strb.Clear()
Application.DoEvents()


Return Nothing
End Function

Private Sub StartCompression()
Dim messblocks As Integer = 1
If calculatedblocks = 1 Then sizeblock = Lastblocksize Else sizeblock = Mem
Dim offset As String = "&H0" '/Input file offset
Dim blockcouter As Integer = calculatedblocks
strb.AppendLine()
strb.Append("Compression start............" & blockstartpos & vbNewLine)
logMessage(strb.ToString)
strb.Clear()

Do Until endfile = True
Dim MemBuffer As Byte()
MemBuffer = ReadBufferBytes(InputPath, offset, sizeblock)
Dim compressedbuffer As Byte() = New Byte(MemBuffer.Length) {}
Dim compressedsize As UInteger = 0
strb.Append("Compress block (" & messblocks & ")" & vbNewLine)
logMessage(strb.ToString)
strb.Clear()
strb.Append("Compress buffer bytes:---> " & "0%" & vbNewLine)
logMessage(strb.ToString)
strb.Clear()
'/ senden an Compressor
For i As Long = 0 To sizeblock
compressedsize = lzo_compress(MemBuffer, MemBuffer.Length, compressedbuffer, compressedbuffer.Length, _
CUInt(CompressionMethodscompress(xb360Generic.Comb oBox2.Text, True)), xb360Generic.levelNum.Value)
'/ update bei 1 Kb oder ende sizeblock
If i Mod 1024 = 0 Or i Mod sizeblock = 0 Then
fortschrittrb(i, sizeblock)

End If
Next
strb.Append("Compressed block size: " & compressedsize & "(bytes)" & vbNewLine)
strb.Append("Clear Memory buffer: " & MemBuffer.Length & "(bytes)" & vbNewLine)
strb.AppendLine()
logMessage(strb.ToString)
strb.Clear()

'/Lösche memory array von Compressor
Array.Clear(MemBuffer, 0, MemBuffer.Length)
'/check ob es was zu schreiben gibt
WriteBufferBytes(OutputPath, offset, compressedbuffer, compressedsize)
'/Nach schreiben array löschen
Array.Clear(compressedbuffer, 0, compressedbuffer.Length)
'/offset für writer setzen next block
blockstartpos = "&H" & CStr(Hex$(blockstartpos) + Hex$(compressedsize))
'/offset für next block berechnen
offset = offset + CUInt(sizeblock)
offset = "&H" & Hex$(offset)
'/block counten
messblocks += 1
blockcouter -= 1
'/Check ob letzter block erreicht ist und loop ende
If blockcouter = 0 Then Exit Do : endfile = True
'/ o vor 1 verhindert das sizeblock berechnet wird bei nur 1 block
If blockcouter = 1 Then sizeblock = Lastblocksize
Loop

End Sub



Private Function WriteBufferBytes(OutputPath As String, offset As String, compressedbuffer As Byte(), compressedsize As UInteger)

strb.Append("Write Compressed block: " & compressedsize & "(bytes)" & vbNewLine)
strb.AppendLine()
logMessage(strb.ToString)
strb.Clear()
Using fs As New FileStream(OutputPath, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Write)
'/check ob file header da ist

Dim headerbuffer As Byte() = New Byte(Headersize - 1) {}
bw = New BinaryWriter(fs)
'/ check ob header benötigt wird für multi blocks durch boolen wert
If multiblocks = True Then

If positionheader = "&H0" Then
strb.Append("Create Big File Header: " & Headersize & "(bytes)" & vbNewLine)
strb.AppendLine()
logMessage(strb.ToString)
strb.Clear()

'/erstelle header
fs.Seek(0, SeekOrigin.Begin)
fs.Write(headerbuffer, 0, headerbuffer.Length)
fs.Seek(0, SeekOrigin.Begin)
WriteInt16(calculatedblocks)
WriteInt16(CDec(512))
WriteInt16(Headersize) '/ siehe header structur
WriteUInt32(compressedsize)

Array.Clear(headerbuffer, 0, headerbuffer.Length)

positionheader = "&H" & CStr(Hex$(positionheader) + Hex$(10))
Else
strb.Append("Update Big File Header: offset " & positionheader & vbNewLine)
strb.AppendLine()
logMessage(strb.ToString)
strb.Clear()
'/ Update header
fs.Position = positionheader
WriteUInt32(compressedsize)

End If

End If
'/schreibe die daten
fs.Seek(blockstartpos, SeekOrigin.Begin)
fs.Write(compressedbuffer, 0, compressedsize)

End Using

Return Nothing
End Function

Private Sub logMessage(ByVal message As String)
With xb360Generic.RichTextBox2
.AppendText(message)
.Refresh()
.ScrollToCaret()
End With
End Sub

Private Enum Endian
Little = 0
Big = 1
End Enum

Public Sub Write(value As Byte)
bw.Write(value)
End Sub
Public Sub WriteInt16(value As Short)
bw.Write(value)
End Sub
Public Sub WriteInt16(value As Short, type As Integer)
Dim buffer As Byte() = BitConverter.GetBytes(value)
If type = Endian.Big Then
Array.Reverse(buffer)
End If
bw.Write(value)
End Sub
Public Sub WriteUInt16(value As UShort)
bw.Write(value)
End Sub
Public Sub WriteUInt16(value As UShort, type As Integer)
Dim buffer As Byte() = BitConverter.GetBytes(value)
If type = Endian.Big Then
Array.Reverse(buffer)
End If
bw.Write(value)
End Sub
Public Sub WriteInt32(value As Integer)
Dim buffer As Byte() = BitConverter.GetBytes(value)
bw.Write(buffer)
End Sub
Private Sub WriteInt32(value As Integer, type As Integer)
Dim buffer As Byte() = BitConverter.GetBytes(value)
If type = Endian.Big Then
Array.Reverse(buffer)
End If
bw.Write(buffer)
End Sub
Public Sub WriteUInt32(value As UInteger)
Dim buffer As Byte() = BitConverter.GetBytes(value)
bw.Write(buffer)
End Sub
Private Sub WriteUInt32(value As UInteger, type As Integer)
Dim buffer As Byte() = BitConverter.GetBytes(value)
If type = Endian.Big Then
Array.Reverse(buffer)
End If
bw.Write(buffer)
End Sub

End Module

Vulnavia
08-14-2013, 04:11 AM
Does it actually work now?

Because i imported it into Delphi to use it for a Save which seem to be compressed, but all it does is copy IN to OUT, no matter what Compression Method i select

It acts like Memcopy, but does not compress anything and i fill the out buffer with $0 so its certainly the DLL that copy the stuff over.

Would be nice to get it work, i don't know if it will support the compression of the save, but well if it does not even compress anything, i don't even try to decompress something... lol