Feature Request: --include / --include-from

[brain-freeze]

Hey,
I switched to gocryptfs from encfs and it works great so far. However the performance over sshfs is much worse than it was with encfs (and #35 is still a problem). So I thought reverse mode may be the better idea for backup use cases anyways. The problem here is a missing --include/--include-from option, which works just like the parameter for rsync. This would allow it to create an encrypted view of certain folders instead of an entire directory.

[rfjakob]

Hi! Thanks for the report - what version do you run and how much worse is "much worse"?

[brain-freeze]

I don't have some exact numbers, but I have the impression that it's related to directories with many small files. I will do some testing with this and report back.

[brain-freeze]

The version was 1.4.3, but I couldn't reproduce any performance shortcomings compared to encfs. Probably I just got the impression after the replacement of encfs, but there isn't any real issue here.
Anyways, it would be great to get rid of that sshfs layer, because this definitely slows down the data transfer rate.

[brain-freeze]

Lately I switched to reverse mode for better performance with remote syncing. An "exclude-from" command would come in here very handy, too. This could save some time to avoid syncing cache folders or VMs that I don't need backup for.

[charles-dyfis-net]

Might I suggest bind mounts to map the directories you want to exclude through from the underlying storage?

[brain-freeze]

How would you do an exclude with bind mount? I'm using https://github.com/gburca/rofs-filtered for now.

[charles-dyfis-net]

mount --bind /unencrypted/storage/thing-that-does-not-need-encryption \
             /encrypted/mountpoint/thing-that-does-not-need-encryption

[rfjakob]

@charles-dyfis-net: interesting idea!

I think brain-freeze wanted to not copy files at all, not only not encrypt them. I guess you could bind mount an empty folder over the folder you want to exclude?

[brain-freeze]

The --include-from/--exclude-from feature of rsync is crucial for my backup purposes. With reverse mode of gocryptfs it's not possible to use this anymore, because the file/folder names are encrypted as well. The only possibilities to overcome this, which are working with folders and files aswell, are (afaik):

• Don't use reverse mode, but regular mode with underlying sshfs. This approach has really bad performance with my internet connection.
• Filtering on the file system layer before data gets encrypted. I discovered rofs-filtered lately and it's exactly providing this feature for a read only view of a folder. For now it looks like if it's doing the job as expected, so that is a viable workaround. However it would be great to have this directly in gocryptfs at some point.

[rfjakob]

Option number 3 is to get the encrypted file names by searching for the inode number (find -inum) or with https://github.com/rfjakob/gocryptfs/blob/master/contrib/ctlsock-encrypt.bash

But i agree that --exclude would be much more user friendly. I'll see how this could be implemented. Problem is that the stdlib does not support passing multiple --exclude arguments. I would have to use another argument parsing library.

[rfjakob]

PS: This is how the backintime backup tool handles exclusions with encfs reverse mode.

[rfjakob]

PPS: looks like cobra is the cli library of choice nowadays.

[brain-freeze]

Personally I think that --exclude-from, where you can specify a file containing all paths to exclude, is a bit more useful anyways. If you want to exclude stuff, it's likely that you want to exclude more than one folder/file. If you do this with multiple --exclude arguments you end up with a messy command line pretty fast.

[rfjakob]

I have just added an --exclude feature to reverse mode. Please test!

I have chosen --exclude over --exclude-from because it is more generic. You can still put your excludes into a file like this:

gocryptfs -reverse $(cat exclude.txt) /home/user /mnt/user.encrypted

With exclude.txt containing lines like this:

--exclude Movies
--exclude Music
--exclude "Ebook Collection"