A ChiFS Hub has generally two ways to index a Share:

1. Delete and reindex the Share from scratch - this causes heavy disk I/O, but is otherwise quite efficient.

2. Compare the current state of the database with the new state of the Share and change only what is needed. This significantly reduces disk I/O if not much has changed (very likely), but at the cost of using quite a lot of memory to do the comparison. In the order of ~500MB per 1 million files.

Ugh, choices.

By using checksums for comparison, a horrible Rust hack to construct memory-efficient structs, and reading the Share index twice, I can get the memory requirements for solution 2 down to ~136MB per million files. That's much better, but I'm not sure it's worth it...

And behold said "horrible Rust hack".

: "You can only safely construct a DST by coercing struct with a size known at compile time"
Me: "Behold my copy-paste power!"

It goes on for 1000 lines. I really ought to be expelled from the programming community for this abomination.

@ayo Yeah, looks ugly 😆 I don't have enough context... Why not using `path: Vec<u8>`??? If you really want something like this: a macro could help 😉.

@t0k A Vec does the same thing, indeed, but requires considerably more memory. What I'm doing here is storing the data inside the same allocation as the struct, akin to what in C is known as the "struct hack" or in C99, "flexible array member".

Sign in to participate in the conversation

We are a cute and loving international community O(≧▽≦)O !