lg-lbf-extractor

Soooooo since my LG G4 (H815) rip'd hard for the second time (first time was due to the famous "big cores" bootloop issue), I had to revive it one more time in order to make a final backup of the internal storage. Most of my shit was on the external but contacts, SMS history and some application data are always stored on internal. ;_; I figured the easiest way to get "everything" off it was to use an LG proprietary tool, aka LG Bridge. I set up everything so if my phone was actually revived by baking it in the oven (7 minutes @ 190C in a pre-heated oven, which may be hazardous to your health lol), I could quickly make the backup and not care if it dies agen. =]]] It did boot up, otherwise I wouldn't have made these skrips for extracting such a shit format backup. xd

Background info:
Cuz it's a proprietary (i.e. shit) format, I'll explain a little bit of the makeup of the LBF file (or what I figured out so far lmao).

• The LBF backup is about the same size as the user-generated data on the internal SD card (application data, pictures, etc), these are all compressed and/or concatenated into one file. LG seems to like fucking up headers/footers in the process. Someone said it's double compression but I tried a bunch of shit and it wassa no-go, not mentioning the fact that whatever is extracted has single compression (ZIP or USTAR, not ZIP+USTAR).
  
• In my case there were 54 ZIP files containing useful data (SMS/MMS, contacts DB, some LG shit like lockscreen settings, etc) inside it. There were a ton of other ZIPs but they only contained application images such as launcher pics, or simply metadata/manifests etc. Many of these also had messed up headers and footers so you'd get a ton of files of roughly the same size containing the same shit, wasting a ton of disk space. >_>
  
• There are also USTAR-format files after all the ZIPs (59166 for me) which are also broken; their footers don't match what the spec says (1024 null bytes) and it might be missing some bytes in the header too. These contain a bunch of directories from /data/data which house your applications' run-time data. I was particularly interested in getting my Swype dictionary from here. =]
  
• The /storage/self/primary directory should be contained somewhere in here too (i.e. your homedir, has Downloads, Pictures, etc). I have yet to find it though. ;_;
  
  

Requirements:

• At least Python 3 y0 (for lbf2tar.py obv)
  
• Any version of Perl should do as long as you have DBI + DBD::SQLite, Getopt::Long and File::Path (to retrieve your call log m8)
  
• Obviously some sort of l00nerx/BSD system to run the Bash script
  
• SQLite reader, iirc Firefucks has an addon for it (or just use the sqlite3 CLI program)
  
  

Notes:
You may see (a lot of) messages like:

tar: (Empty error message)
tar: Error exit delayed from previous errors.

Apparently this is a bug in certain BSD flavours of tar, it seems to extract shit just fine though. GNU tar (gtar, gnutar) may or may not complain about it not being a tar archive, in that case maybe try 7-Zip (7z) or unzip. =]

You may also see some of the following messages:

Truncated ZIP file body
ZIP compressed data is wrong size
Bad ZIP file

Once again it seems to extract fine (up to the point of error) and in my case happened maybe 5 times throughout the entire backup file. Most of em were binary files so idc. =]

The USTAR files also seem to throw errors when decompressing core dumps. You don't really want this on a new device anyways so I'd say it's safe to ignore. ;]

Usage

First step:
./lbf2tar.py <LGBackup*.lbf file path> <extraction directory>

The Python script starts by counting the amount of special file headers found in the LG .lbf file. Each of those is extracted to a zero-padded numbered .tar file inside the specified extraction directory (tar > unzip for broken archives imo tbh phambly).

You may need to change the shebang line of the .py script (#!/usr/local/bin/python3) if your Python3 is installed elsewhere.

The script also contains a line DEBUG = False, which makes it do up to N ZIPs/USTARs only. I used this for myself for quick testing so it's not an argument since you prolly won't be needing it.

Second step:
./extractem.sh [-hv] [-dnts]] <extraction directory> <final destination>

Options (these must be specified before the required arguments):

• -h: show built-in halp
  
• -v: show version
  
• -d: delete tar-extracted directories as we rsync (otherwise you could have one directory with 50k+ of 'em lol)
  
• -t: delete .tar files as we go (same note as -d)
  
• -n: don't rsync, just extract the tars as-is (speeds shit up)
  
• -s: silent (no -v flags for rsync/tar commands inside the loop)
  
  

After running teh pythinz, <extraction directory> should be filled with .tar files. These need to be extracted into <final destination> so you can fucking finally access your home made pr0nz very important files.

Every .tar file is extracted to a separate directory before being fed to rsync (unless -n is used obv). I'm using rsync because that means your OS won't have to copy duplicate files over and over (thereby raping your disk a bit). If no relevant deletion flag is specified, will create a directory _DELME_DIR inside <extraction directory> to store the numbered dirs and _DELME_TAR for the tar files.

When you can finally access dem files, you'll notice a bunch of .db ones. You can open these with any SQLite reader to at least view em data. =]

(Optional) third step (prolly need r00t tho lol):
I also wanted to restore my call logs, so I wrote another skrip to convert an old-style contacts2.db to the newer calllog.db format. On my new phone (Android 8.1/Oreo) I had both those files so it seems they now store call logs separately from your contacts etc. Which I suppose is a good thing cuz it would make restoring that shit that much easier in the future. >_> This contacts2.db file can be found in <final destination>/data/user/0/com.android.providers.contacts/databases after having done the second step. If it already contains a calllog.db file then just skip to the end of dis dere section. ;]

With a little bit of luck you can simply restore the contacts2.db with commands similar to those in the last code section, but you will prolly end up with a shit tonne of dupes.

I filed this skrip under the LG restoration process cuz it was my reason for writing it. Also many people would probably be using an Android version that uses the new format by now anyways. xd

So:
./demcalls.pl [-h|--halp|--help] [-u=<0/1>|--add-for-all-users=<0/1>] <contacts2.db file path> <calllog.db output directory>

Options:

• -h, --halp, --help: all do the same thing, namely shitting out the built-in halp text =]
  
• -u=<0/1>, --add-for-all-users=<0/1>: set add_for_all_users column to specified value (defaults to 1)
  
  

This skr1p also has a $DEBUG flag which is set to 1 by default. The only thing it does so far is printing the relevant query if it failed to INSERT for whatever reason. xd

You'll prolly wanna look inside /data/user/0/com.android.providers.contacts/databases on your device. Leave the contacts2.db* and profile.db* files alone and remove/rename both calllog.db and its journal file. Copy the restored .db over to that folder and make sure to set the proper permissions:

PL2_sprout:/data/user/0/com.android.providers.contacts/databases # ls -la *.db
-rwxr-x--x 1 root   root    335872 2018-08-07 20:54 calllog.db
-rw-rw---- 1 u0_a76 u0_a76 1458176 2018-08-07 20:45 contacts2.db
-rw-rw---- 1 u0_a76 u0_a76  397312 2018-08-07 20:31 profile.db
# Notice how both the permissions and ownership are fucked?
PL2_sprout:/data/user/0/com.android.providers.contacts/databases # chown u0_a76:u0_a76 calllog.db
PL2_sprout:/data/user/0/com.android.providers.contacts/databases # chmod 660 calllog.db
PL2_sprout:/data/user/0/com.android.providers.contacts/databases # ls -la calllog.db
-rw-rw---- 1 u0_a76 u0_a76 335872 2018-08-07 20:54 calllog.db
# We good m9

One final thing: unless you've disabled it you're probably running SELinux in enforcing mode, as such you'll have to restore security contexts for that shit too (restorecon -F calllog.db).

In my case I also had to clear the app cache for both my Phone and Contacts apps (just go through Settings > Apps > etc), otherwise they would crash on open cuz they cached something that's not (yet) present in the generated .db. =]

Ayy lmao
Whatever comes after that would be an exercise for the reader L M A O