PDA

View Full Version : First public release of Recom (A comprehensive and superior compression format)



qriist
01-07-2012, 02:16 AM
This is a toolset I have been working on for quite some time. The basic premise is very simple and adheres to two points:
A: The files must be compressed to the smallest possible file size in the most reasonable amount of time.
B: The file integrity must be protected at all costs!!!

I am a semi-amateur programmer and have written these tools in AutoHotKey_L 64-bit. I have tested over and over using many different scenarios. Alas, I can't predict everything. This release contains the first module for Recom that I will be releasing, Revolution. Revolution processes Gamecube and Wii games.

Main features:
-Automatic! You place your game images and archives in the indicated directory and Recom will process it in turn. Split RAR's and binary splits are no problem!
-Safe! At no point will your original image be deleted until all stages of compression and other forms of processing complete successfully.
-Efficient! I have spent a large amount of time messing with these tools to find the fastest, best, reproducible settings imaginable.
-Naming! If the option is enabled and a ReDump dat is available, the program will rename your output game based on the dat file at hand, with semi-auto updating of the dat files. (Once ReDump restores the dat files to the website I can make it entirely automatic..) This allows for standardized naming conventions and easy updating of already processed files. (Renaming is much faster.)
-Unified! For images with garbage data, this data is moved to an external file. Users can opt to download only the scrubbed archive (or the scrubbed archive+junk data) and the unRecom extractor will detect and use the proper technique. This means that "casual" users can easily grab the tiny scrubbed archive while the "collectors" acquire the entire thing, and both seed the same torrent!
-Economic! If everyone uses this method, even those with slow connections or limited space will be able to download more games in less time.
-Futureproof! If an amazing new compression format or technique pops up, I'll be able to implement it and the end user will be able to run all previously Recom'd games through it.
While the script looks quite primitive and unrefined, I do intend to add a better visual interface if the community likes what they see. (That is, if I get good reception on the METHODS at hand.)

This software is still BETA, though I have processed over a hundred gamecube and wii images. I can vouch that all characters A-Z, 0-9, parentheses, and the ampersand (&) are properly picked up while renaming. If other special characters are found in the redump dats that use special codes, I don't know them. The program will sorta fritz in this case... Just kill the plugin (ex, "Revolution.exe") and inform me of the game and system. Your original game will NOT be touched, mmmkay?

User feedback prompted me to make an intrinsically dynamic system. You need to run the Recom Initialization program before first run. Your "bit-ness" and RAM capacity will be detected and Recom will automatically compile the best possible settings for you! The meta files will allow your instance of Recom to "communicate" with previous instances.

Good luck! I hope you guys start using this format from here on out!

All I ask for is feedback. (Does it work? does a Certain game crash it? Using too much RAM? Missing some real feature?) The readme follows, and you should read it (lots of info) but first, some numbers.
Gamecube images are ALWAYS 1.35gb
Wii images are ALWAYS 4.37gb/7.92gb single/dual-layer


The following are some sample results from my UNTOUCHED, INTACT ISOs, compressed with my format. In red following each game, it shows what the scrubbed ISO + winrar looks like, size wise.


The Legend of Zelda - Skyward Sword (Korea) 3gb (ISO+RAR=3.83gb)
Rayman Origins (USA): 1.44gb (ISO+RAR=3.44GB)
Super Smash Bros. Brawl (USA) (v1.01) 5.11gb (ISO+RAR=6.73GB)
Geist (Europe) 340mb (ISO+RAR=647mb)
Metroid Prime Trilogy: 4.41gb (ISO+RAR=7.43gb)


Granted, not all results are as impressive, but the format is undeniably a step forward in better compression schemes.



Recom by Qriist


"Recom" is a toolset that aims for the maximum amount of compression on supported game types, with a secondary focus on making damn sure your data is intact. Many tools do not check file integrity after compressing or when moving files from one drive to another, a serious flaw that should be addressed! This is my brainchild, as a result. ALL data-intensive operations are verified against the desired and actual outcome and redone until the computer gets it right.


For the most effective results you will need at least 12gb of RAM. Sorry, that's just the way it is, blame 7-zip. To that end, however, I have created a solution that allows ALL XP and higher users the ability to automatically detect and set your BEST LZMA value. This value is then used in future Recom processes to determine if you can (possibly) further compress this particular file. Once again, for best results, you need a fairly strong computer, though CPU is much less of a concern. This may seem hefty, keep in mind that your storage considerations will decrease, sometimes dramatically.


Programming for the Recom toolset (includes Recom.exe and all Recom plugins) is done in AutoHotKey by a self-taught programmer, though I cannot speak for other used tools.


Once extracted, inside the Recom folder, you will see a minimum of 4 folders, Recom.exe, Recom.ahk, Recom.ini, and this readme. Here is a quick rundown of the functions:
-Recom Initilization.exe: This program compiles all other plugins with the correct detected settings. You must run it once and wait for the success message.
-Recom.ahk: This is the launcher program, with .ahk being the source code. This isn't very advanced right now, little more than a generic loop for the plugins, but I do plan to add a few things to it.
-Recom.ini: Basic config file. Very easy to understand.
-(Folder) RecomPlugins: This is the folder that Recom will run through, launching each one of these programs in turn. If you are perhaps dabbling in what makes Recom tick, you will like to know that the source for each plugin is included; additionally, if you launch Recom.ahk (Requires AutoHotKey installed!) then all plugins will run as source, the AHK (NOT the .exe) version of each plugin. Good for debugging changes. Each plugin can be run independently.
-(Folder) RecomTemp: This folder is where all the magic actually happens. If you kill Recom intentionally or unintentionally you may safely delete anything in this folder. Recom will start from scratch anyways.
-(Folder) RecomHold: While you can specify your own output directory in the Recom.ini, all completed files will default to (Recom)\RecomHold\(system)\(Relative dir). This means that relative folder structure is kept 100% intact while moving your heavily compressed files safely away from the scope of the Recom process, with the sole exception of Alias.
-(Folder) RecomTools: This folder contains all the tools used by any plugin of Recom. All tools are freeware or shareware, so there is no need to crack anything. While it is true that most shareware programs are crippleware, my implementation of such tools will NEVER fail due to such limitations. I make sure that each tool is not crippled in what it needs to do.






[Getting Started]
Extract the latest Recom into a folder of your choosing and you'll see all of the items described above, as well as some folders that reflect default settings of the currently released plugins. Grab a copy of the latest redump zip, currently located here: ( read the readme ), and run the "Recom Initialization.exe" to analyze your system and compile all of your plugins with the right settings. Put some games in the appropriate folders and run Recom.exe. You will get a notice about the dats, which can take a fair amount of time to go away, usually no more than ten minutes. Then, your games will be processed according to all of your settings, most of which should be at their defaults.






[Supported formats]
===The following input archives are currently supported.
Binary splits (.001)
7-zip (.7z)
RAR
Split RARs (x.r00 AND x.part1.rar)
...Many more to come!




===Supported Wii/GC specific inputs.
.wia
.ciso
.wbi
.wdf
.gcm




===Supported (mainly) PSP inputs
.jso
.dax
.cso






[Currently implented/released plugins]




===Revolution
Revolution addresses Gamecube and Wii games. Because these are so very similar in structure, the fabulous Wit is capable of handling both types. There is a lot that goes into processing GC/Wii games and the most strenuous requirement is harddrive space. To process any one game you will need about 5x the size of the ISO itself free on your Recom drive. Not too big of a deal with GameCube games, but certainly a big requirement with Dual-Layer Wii games. Fortunately, there will be an initial free space check to make sure all is well before the real meat of the operation starts. (NOT YET IMPLEMENTED, SORRY!) As for the process itself, this plugin will start with an original ISO and copy it to WIA/NONE format using Wit. The WIA will then be copied out to an uncompressed scrubbed ISO. At this point, a quick test will be made to make sure that the new ISO is properly created. A delta patch will be made between the newly made scrubbed ISO and the original ISO. This delta patch will be applied to the scrubbed ISO and this NEW ISO will be binary compared against the very first ISO, the original. Assuming all is well the two temp ISOs will be nixed and 7z will do its magic on the WIA. Once the 7z is produced and verified, the 7z and the xdelta will be kept and moved to the output folder.


===Alias
Alias is a plugin that will keep your games named in compliance with your ReDump dats. The plugin reads the folders and dats as specified in the Recom Ini and runs through all of the meta files in your output directory to find games that have a hash match but not a title match; Alias will move such games and all related files to that game's sponsor plugin folder and then run the plugin in question. This is the ONLY plugin that will ever read your finished games and even then it doesn't modify them, it just calls the right plugin. Safety of your data is paramount!






[Ini Options Explained]
[Recom]
MoveFinishedFilesTo= Specify a valid directory and you can output the relative game structure to this new root folder. Otherwise, RecomHold will be utilized. Note that %system% can be used as a folder variable in the path; this will pick up the "root" folder of your input games and place that "root" folder at %system%.
Example: Z:Video Games\%system%\Games <- This will become:
Z:Video Games\Gamecube\Games <- or whatever "root" folder your input games reside in.
DebugMode= if the word "active" is specified, 7zip will use copy mode, greatly reducing the time required to analyze problems in the input files.
WindowMode= the words min,max,hide will cause the console window for each program to do just that. If no word is specified the console window will pop up as normal.
Torrent Creation Mode= This option lets you create or not create torrents of your finished games with the following keywords: all, ReDump, none. These options allow you to always create torrents, create torrents only if your game matchs a ReDump dat entry, or never create torrents.


[Open Torrent Trackers]
(broken at the moment)


[Private Torrent Trackers] ; list one tracker per line to create a torrent for that tracker. Key names under this section don't matter.
Tracker1=read the readme
Tracker2=


[Folders] & [Dats]
Specify the exact name of the plugin for these entries, and also the exact folders and dats to look in. The folders you reference are folders that are present inside Recom root directory. In the case of the dats, do NOT include the timestamp. As an example, if you have "Nintendo - Wii (20111222 06-15-24).dat", only include "Nintendo - Wii" so that future updated dats will be automatically selected.




[Advanced]
Symbol Substitution= These are symbols that will be replaced with spaces in the ini file, so required due to the nature of some of the commands in the ReDump routines. If you encounter a name error (big message box with a load of programming jibberish,) just attempt to find the ReDump name and and that symbol to this list. Run Recom again and all should be fine. REPORT THESE TO ME, PLEASE! I can include such symbols in the next release.
Active Plugins= List the name of the plugins you want ran. If you don't want one to run automatically, ever, just remove it from this list.


Recom: http://www.multiupload.com/RNY4W8BIA8


Thank you.



PS:
I have updated Recom to fix a very stupid mistake.

=2012-01-08=
[Recom] Derp: didn't include the updated Recom Initialization.

garyopa
01-07-2012, 01:46 PM
Many thanks for posting the 'release' here. -- I have now 'front-paged' the news story, and for now moved the thread to the Wii section as it is more fitting.

billysastard
01-07-2012, 01:59 PM
Many thanks for posting the 'release' here. -- I have now 'front-paged' the news story, and for now moved the thread to the Wii section as it is more fitting.

we are sure this isnt a trojan / virus arent we?

i ask because this appears to kick even 7zip's butt in a huge way

qriist
01-07-2012, 09:54 PM
If you have any doubts about my programming, feel free to compile it yourself. I only chain the tools, and thus only take credit for the .ahk scripts and resultant .exe's. I actually USE these scripts so I can assure you there are no virii lurking among the RecomTools files.

billysastard
01-08-2012, 02:26 AM
If you have any doubts about my programming, feel free to compile it yourself. I only chain the tools, and thus only take credit for the .ahk scripts and resultant .exe's. I actually USE these scripts so I can assure you there are no virii lurking among the RecomTools files.

really, i dont, this sounds absolutely amazing and frankly if it takes off it could be a world changer in compression at least in the emulation world, however i was only asking and i'm sure you'll appreciate given our long and sad history with spam and such, much of which is designed to either steal credit card numbers or sell them (and of course counterfeit goods) that when someone with a single post is offering an executable download then its my duty to question it.

no malice was in my question.

DjoeN
01-08-2012, 04:26 AM
I get an error while running "Recom Initilization.exe"


#include file "E:\Recom\RecomPlugins\\RecomPlugins\lib\Ini Library.ahk cannot be opened.

I did change the line32 to: RunWait, %RECOM%\RecomPlugins\%A_loopfield%.%ahk% before running it.
I do get an "Recom.exe" and "UnRecom.exe", but the Recom.exe does nothing when started.

I installed all in the root of my "E" drive , Here's how it looks like:
- E:\Recom\
- E:\Recom\GameCube
- E:\Recom\RecomHold
- E:\Recom\RecomPlugins
- E:\Recom\RecomTemp
- E:\Recom\RecomTools
- E:\Recom\RecomTorrents
- E:\Recom\Wii
- E:\Recom\UnRecom.exe
- E:\Recom\Recom.exe
- E:\Recom\Recom.ahk
- E:\Recom\Recom Initilization.exe
- E:\Recom\Recom Readme.txt
- E:\Recom\Recom Changelog.txt
- E:\Recom\Recom Initilization.ahk
- E:\Recom\Recom.ini
- E:\Recom\UnRecom.ahk

EDIT:
I fixed it by manually editting Redcom.ahk:

Original lines:

#include %a_scriptdir%\RecomPlugins\lib\Ini Library.ahk
#include %a_scriptdir%\RecomPlugins\lib\TF.ahk

Editted:

- #include e:\Recom\RecomPlugins\lib\Ini Library.ahk
- #include e:\Recom\RecomPlugins\lib\TF.ahk

Still Recom.exe does nothing (UnRecom opens a File Browser Window)
Even running it as administrator under Windows 7 Enterpise (64Bit)

Note: For those who think i run everything, I run everything i download first in a sandbox :p

qriist
01-08-2012, 08:01 AM
@billysastard, all good points. :) Caution is always good when using new programs. I was a tad snarky, my apologizes.

@DjoeN, there were a few issues that I had to resolve with the compiled versions. In my own testing I always use the scripts as run natively via AHK. I can't expect everyone to have to download AHK to use this and it requires different settings for 32-bit or 64-bit anyways. Thus, it actually compiles anew whenever you run the Initialization. Every once in a while I forget to recompile the Initialization. All of that said, I've updated it, properly. Link is in the first post.

As far as "it does nothing" goes:


Put some games in the appropriate folders and run Recom.exe.

You gotta drop a game or two into Recom\Wii\ or Recom\GameCube\ before you run it or it does nothing. Read the manual thoroughly. It has a ton of need-to-know info. This isn't a program designed for a couple one-off operations, this program is designed to perfectly compress all of your games in a flawless manner with no user interaction. Maybe later I will add some examples as explanation.

Also, I updated the first post to show what the SCRUBBED ISO's look like compared to my INTACT ISO's.

billysastard
01-08-2012, 08:51 AM
@billysastard, all good points. :) Caution is always good when using new programs. I was a tad snarky, my apologizes.

no worries.

somenetdude
01-08-2012, 02:12 PM
extracted, ran it once says good to go, now try to run exe and nothing.


windows 7 ultimate
amd phenom quad 2.5ghz
8 gigs ram

looks and sounds cool..

qriist
01-08-2012, 03:21 PM
extracted, ran it once says good to go, now try to run exe and nothing.

Read the readme, it says you need to place your ISO into one of the relevant folders. It can be in a RAR or 7z, or presented as any one of the formats as listed in the readme.

qriist
01-15-2012, 01:53 PM
Updated with a few features.