Tools for scanning dd images / Finding an encrypted file

I am not at all familiar with Brtfs, but from a quick look at it specs, it seems like the main Superblock has mirrors, have you checked if (by sheer luck) they can be found?
https://btrfs.wiki.kernel.org/index.php/On-disk_Format

Maybe, if you can restore the main superblock, then tools *like*
https://github.com/knorrie/python-btrfs
https://github.com/shujianyang/btrForensics

May be capable of reading the tree/subtree.

About hex editors, if you are OK with using Windows, the good ol'TinyHexer can access sector by sector an image that is mounted as disk.

I believe that a number of disk editors behave the same, but you need to "mount" the image as (RAW) disk device (or get another hard disk and dd the image to it, then access the \\.\PhysicalDrive).

Still 00'ing (or FF'ing) anything that was recognized in Photorec or DMDE may prove useful.

jaclaz