By using checksums for comparison, a horrible Rust hack to construct memory-efficient structs, and reading the Share index twice, I can get the memory requirements for solution 2 down to ~136MB per million files. That's much better, but I'm not sure it's worth it...
And behold said "horrible Rust hack".
#Rust: "You can only safely construct a DST by coercing struct with a size known at compile time"
Me: "Behold my copy-paste power!"
It goes on for 1000 lines. I really ought to be expelled from the programming community for this abomination.
@ayo Yeah, looks ugly 😆 I don't have enough context... Why not using `path: Vec<u8>`??? If you really want something like this: a macro could help 😉.
@t0k A Vec does the same thing, indeed, but requires considerably more memory. What I'm doing here is storing the data inside the same allocation as the struct, akin to what in C is known as the "struct hack" or in C99, "flexible array member".
We are a cute and loving international community Ｏ(≧▽≦)Ｏ !