Thanks for verifying my somewhat out of date guesses about smudge performance!
Re the python startup time, this is particularly important for smudge/clean filters because git execs the command once per file that's being checked out (for example). I suppose even go/haskell would be a little too slow starting when checking out something like the 100k file repos some git-annex users have. ;)
Yep. It's really a problem that needs to be fixed in git proper. I'm surpised that github of all people didn't realize this and/or invest the time to do it right.
The one major obvious drawback for it being fixed in git is that it's not backwards compatible with old clients though. Doing it in go is probably a good improvement over the existing solutions of git-media and git-fat but I don't think is the final one.
Funny enough, although I had thought this since I started working with git-fat, I only recently admitted it[1]. Perhaps if I had admitted it when I first started work on it then there's a chance they would have seen it! :-P
Ok, that means I don't have to check out git-lfs, git-fat or git-bigstore. My annex is 250k symlinks pointing to 250 GiB of data. It's slow enough as it is.
At 250k files in one branch, you are starting to run into other scalability limits in git too, like the innefficient method it uses to update .git/index (rewriting the whole thing).
There is the new "split-index" mode to avoid this (see "git update-index" man page). The base index will contains 250k files but .git/index only contains entries you update, should be a lot fewer than 250k files.
Re the python startup time, this is particularly important for smudge/clean filters because git execs the command once per file that's being checked out (for example). I suppose even go/haskell would be a little too slow starting when checking out something like the 100k file repos some git-annex users have. ;)