dehydrate-fs/README.md

# <center>Dehydrate-fs (Beta)</center>
`dehydrate-fs` is a family of tools for separating out files from disk images for the efficient storage of both. The project currently exists as a minimum viable product supporting only the `ext2/3/4` filesystems.

## Quickstart
```bash
#Generate the filesystem map
map "$file" mapfile.dat

#Dehydrate & compress the filesystem
dehydrate "$file" mapfile.dat | zip -1 "$file".dhd.zip -

#Rehydrate the filesystem
funzip "$file".dhd.zip | rehydrate mapfile.dat "$file".rhd

#Compare results
cmp "$file" "$file".rhd
```

## Installation
The scripts may be ran directly. Please ensure you have `perl` and `e2fsprogs` available.

## Usage
### `map FILE [MAPFILE]`
Create a mapping of files in the partition image and extract their contents. If `MAPFILE` is not specified, the output is written to `STDOUT`. Files are placed in `./pool/` and are named with their `sha256sum`.

`map` accepts an environment variable `THRESHOLD` for minimum filesize bytes. It defaults to `1048576`.

### `dehydrate FILE MAPFILE [OUTPUT]`
Create a copy of `FILE` with zeros written to the locations specified by `MAPFILE`. If `OUTPUT` is not specified, the output is written to `STDOUT`. To prevent terminal corruption, the program will not run if `STDOUT` is a terminal.

It is recommended that you stream the output into a compressed archive as the dehydrated file is the same size as the input. `zip` is recommended, but `xz` performs similarly enough. `gzip` does not appear to be appropriate unless higher-quality compression is desired.

```bash
dehydrate "$file" "$mapfile" | zip -1 "$file".dhd.zip -
```

### `rehydrate MAPFILE [OUTPUT]`
Read from `STDIN`, replacing specified subsections with file data according to `MAPFILE`. `rehydrate` requires that the file contents are available under `./pool/`. If `OUTPUT` is not specified, the output is written to `STDOUT`. To prevent terminal corruption, the program will not run if `STDOUT` is a terminal.

## FAQ:
#### Why is this necessary when chunk-based deduplicating storage systems exist?
To my knowledge, most chunk-based deduplicating storage systems operate at a very coarse level that isn't suitable for collections of average-sized or fragmented files.

#### What is the danger of dataloss?
The tools are written entirely in bash with very little mind paid to error handling. It is assumed the user will verify data integrity before committing irreversible actions. That said, the pseudo-formats are developed specifically to be as simple as possible. Dehydrated `ext2/3/4` filesystem images are mountable using native tools and the mapfile format is trivial to parse.

#### Why the hell is it programmed entirely in bash?!
Because I could.

#### No seriously, why?
I am not a clever man. Even toy programs for interacting with `ext2/3/4` make my head swim. Too many details, not enough visible intent. I prefer shell scripting for this reason.