Trying to be constructive, I would say this needs to zoom out a few layers and make a case for what concepts it is trying to address.
The primary means to distribute open source software is via source code. Automagical technologies to assemble all the source code for a dependency tree, and/or automagical technologies to fetch precompiled dependency trees with curated options, versions, component granularities, etc are extremely non-trivial problems. To date the only examples of this done with a measure of effectiveness are actually entire operating system distributions backed by software like apt, yum, pkg-src, etc.
Primarily because of the total failure in this regard of windows and osx, individual languages themselves starting advancing ecosystem centric dependency management with files retrievable from servers hosted on line. At best, this style strategy solves some of the problems some of the time, and at worst breaks the ability to make things even work the old way - ie manual download, assembly, and compilation of source code dependency trees.
In the context of largely portable, interpreted code, this sort of thing is at plausible. In the context of native code with multi-dimensional matrices of OS/Compiler/libc/versions/flags/phases of the moon, the notion of a repository of precompiled binaries to even enough cover "common cases" to be thought of as a language level feature.
In the case of source dependency management, exposing the information about where and how things are installed and reachable for use in the build environment of the foreground project is hard enough, and trying to be able to hold cross platform build recipes of some sort is literally the creation of yet another operating system. Take homebrew for example.
Perhaps some sort of unification wrapper over apt and pkg-config is a thought, but honestly can't even guess what problems can be addressed with something like this.
In all seriousness the only cross platform solution maintaining build environments is to just host all builds on linux, use mingw64 and osxcross along with apt or yum + regular pkg-config, git, etc.
This looks like someone who thinks things like pip and gem make sense trying to overlay that paradigm to a broader scope. Or maybe I'm just too old to "get it". Please clue me in?
> To date the only examples of this done with a measure of effectiveness are actually entire operating system distributions backed by software like apt, yum, pkg-src, etc.
Rust achieves it. Cargo will download dependencies built for the client's desired platform.
I may be wrong but I think D does it as well.
You are right that C/C++ has a wider range of options available to it, but that is for historical reasons.
The principal benefit of a package manager like this is that it forces the packages contained therein to follow a uniform convention of platforms, build types, and options. C/C++ has been missing this for decades.
Interesting. I'll have to look at this in more depth when I'm off the phone. It does seem a bit over to top.
I've been toying with a similar idea that would focus only on static libraries and would handle binaries as an optimization. Just being able to specify "fetch zlib, protocol buffers and libjpeg" and build them for me would simplify many of my C projects.
The only coherent way I could think of to manage architectural differences was to force a single build tool that understood the important compiler flags and version incompatibilities.
Building from source solves a lot of problems but somehow even today it still takes forever to compile certain projects (looking at you, grpc).
I can't speak for other languages, but for C++, it's mainly the huge build time just to get started using the project. This is obviously an issue for any C++ project, but with GRPC, having to build all of OpenSSL is particularly annoying. (Especially if you're not using secured channels!).
I also couldn't figure out how to build just the C++ static libraries easily. Instead I end up building everything and then deleting the shared libraries. That's easier to do than trying to figure out how to convince the linker that I prefer to link statically.
Once that's out of the way it's pretty reasonable. It took a while to figure out that `--cpp_out` doesn't actually generate stubs, though I'm sure it's documented.
So it's not really worse than any other package; it's just _big_, and that makes it a little more annoying. Having any kind of reasonable package manager for C would simplify things dramatically. Not having to express my dependency on GRPC as a git submodule+some clunky Makefile rules would be nice.
I tried using gRPC with Go before having to abandon the project for reasons outside my control. Looking forward to using it once again and this time take it to production. The only trouble I had was with the documentation which was scarce.
Anyway, here are some comments on how to improve things with gRPC:
1.
A dedicated mini-site for each supported language. This http://www.grpc.io/docs/guides/auth.html is really confusing to read due to the cocktail of languages. All examples need to be in all languages and the user should be able to choose which language to view documentation for.
2.
http://www.grpc.io/docs/guides/error.html needs to be populated. Right now I use a custom error code enum w/ 0 as SUCCESS so that my downstream consumers (app developers) get a descriptive identifier. I don't even know if there is anything else I should be aware of that gRPC offers to work with errors.
3.
A how-to on setting (on a client) and accessing (on a server) custom HTTP headers in gRPC. This is needed for reading the Authorization header during authentication for example. Another example is setting the From header (email of user) and using this to enable persistent sessions. I appreciate the docs covering these two specific examples as they are rather common.
4.
A simple guide on how to manually make an application speak the gRPC protocol - both as a client and as a server. Helpful for those who want to integrate other languages w/ gRPC enabled languages. Endgame is they can speak the wire protocol and understand the conventions just enough to quickly write a working client/server for their app. Great if this example uses JavaScript (most people have to learn it anyway due to the browser).
5.
RPCs with multiple messages as arguments in input and/or output. All examples have SomeServiceRequest and SomeServiceResponse messages for SomeService RPC giving the impression that all RPCs are single argument in the .proto file. This isn't stated explicitly anywhere creating an ambiguity.
6.
Move to Discourse from Google Groups. Much easier to use and would let us organize the discussions better. Also use this to power comments on the gRPC website for each page. I can see this step alone helping the community aspect of gRPC immensely.
7.
Create a visual gRPC explorer. Ideally this should be a simple HTML/CSS/JavaScript static page running on browsers that support HTTP/2 and making it would be a good way to start supporting gRPC on the browser (refer Point 5 above).
"honestly can't even guess what problems can be addressed with something like this"
The problem is that OS package managers install things globally. This is a nightmare when developing, because
* it makes much more difficult to handle multiple versions
* as a consequence, it makes difficult to keep libraries updated, because package authors try to be conservative in what libraries they include in an operating system targeted at non-developers (in particular, latest versions of libraries are always lagging behind upstream)
* it makes it hard to be sure that what you build on one computer will be reproducible on another one
* even in the case where you know which dependencies you need (and you don't forget something just because you had already installed it), there is no simple way to document how to install them in such a way that the next user will be able to install everything with a single command
* it requires root access when it is clearly not needed
> To date the only examples of this done with a measure of effectiveness are actually entire operating system distributions backed by software like apt, yum, pkg-src, etc.
npm seems to handle this just fine for both JS and native (c++) node modules via gyp, and it's officially supported on at least three platforms (Windows, Linux, OS X) using 3 different compilers AFAIK (microsoft c/c++, gcc and clang respectively). Granted, node enforces specific conventions for packaging and lookup (package.json and node_modules), but there's no reason similar conventions can't be created for C/C++ modules, even if they're more complicated due to language specific complexity (dynamic vs static linking). Mantaining packages up to date for a lot of open source projects could be done by a sufficiently dedicated community.
> npm seems to handle this just fine for both JS and native (c++) node modules via gyp
Could you describe this in a bit more detail? Does gyp (in the context of npm) build the binaries at install time, or is it a repository of a bunch of pre-built artifacts?
In my experience of installing npm packages, it doesn't seem like it's building native extensions at install time, but it's hard to say for sure.
It does build them on install time, if a suitable compiler is available. A classic example is the `ws` module (WebSockets), when you install that you'll normally see the build log for its native dependencies on the npm output:
It depends. Someone more experienced may be able to add more info, but gyp appears to know what is available pre-built vs what needs to be built. Depending on what platform and options you've selected it might call (for example) the Visual Studio C compiler to build a native extension for you.
Linux package managers and Homebrew don't solve it, not even close. In fact, their fragmentation makes the problem worse.
If I write a C library and someone wants to use it in their library or application, they have three choices:
1. Add it as a build dependency of their project. This is a huge pain.
- First of all because most build systems are extremely fiddly and poorly designed, so adding the dependency is usually harder than it should be. For example, because CMake, which everyone seems to be using these days, can't produce sensible Visual Studio projects, one project I've worked on maintains separate CMake and Visual Studio build systems - which means that making any moderately complicated changes to the build requires me to either manually edit the VS XML files, send the result to the trybots, and pray it works, or boot a Windows VM. That's just an example; I don't know how representative in the sense that other projects also use that dual build hack, but from struggling with CMake, autoconf, and other build systems, I'd say it's representative in the sense that build systems will always find some stupid way to break and waste your time. Awful. Any alternative is worth consideration.
But let's say we found an alternative that's not a package manager, and ended up with a nice clean system where adding a dependency on a given pkg-config file is just a matter of a few lines and will just work, including for users on weird systems. Yeah.
- Second because the user now has to manually download all the dependencies, and manually upgrade them when they get out of date. Which means you don't want to release too many upgrades, even if you're trying to do rapid development. It's nice that this is possible, I guess, but it's a real pain.
- That is, unless it's in their distribution or system package manager of choice. But going to 10 different Linux distributions, Homebrew, and maybe MacPorts for good measure (and leaving Windows users to rot), and asking each of them nicely to add your library, is not exactly a pleasant experience. Maybe they will add it themselves if an application depends on it, once your library is actually popular rather than a proof of concept. (A bit of a chicken and egg situation.) And then maybe stable Linux users will actually be able to download it in a year or so, but will always be out of date. Of course, if the project using your library wants to be on its bleeding edge, their developers can't do this themselves - they have to go the manual route.
This also doesn't really work for micro-libraries, which I'll elaborate on in a bit.
So it's not surprising that two other approaches are common:
2. "Vendor" your codebase into their codebase, either with something like git-submodule or just by copy and paste. Typically this requires them to figure out how to build your code with their build system. Works okay, but is basically just a poor man's package manager - it has none of the advantages of the previous method, such as systemwide package tracking and upgrades (either by the system package manager or the user manually remembering to do it!), reduction of fragmentation, etc.; and it is more work to maintain.
3. Don't bother with your library at all; just reimplement whatever they want themselves (in a poorer fashion).
It's hard to quantify how much this happens, but in my experience, C++ projects do seem to have a bias toward building up little piles of basic utilities rather than reaching for a library. At the other extreme would be node.js and NPM - this post, as well as a Debian thread from a few months ago it belongs to, containing much handwringing about how to fit it into their system, is a good read:
(Spoiler: It's the build dependency graph for building jQuery, which contains some 491 entries, although it lists packages multiple times when they are referenced by multiple dependencies.)
Some NPM packages are literally 5 lines of code. Arguably still better than copy and paste... maybe...
I don't think C/C++ need to go there; most languages with package managers don't. But it's a good example of how the presence of packaging tools, as internalized into a programming language's culture, significantly changes the way people write code in those languages. And I think C/C++ could be a lot more pleasant if it were easier to reach for utilities outside of the standard library. (Especially pure C, which lacks an equivalent of the STL in its standard library - while the language may prevent writing data structures as neatly as in C++, you can still express some generalized high level structures pretty well and efficiently; and I think the standard library is partly to blame for the fact that people usually don't, instead building half-assed special-purpose data structure implementations every time one is required.)
...
I know, the idea of having a different package manager for every language sounds awful. It is awful, both for the user who needs to remember how each one works, and for the developer who's screwed if they want to depend on something written in a different language. I'd be quite interested in the idea of building something that feels like a language package manager, but is actually just Nix or something with some useful defaults...
But I'm starting to be convinced that having nothing at all is worse.
You allude to the complexity of C builds. While real, a huge fraction of this is not caused by any inherent difference between C and interpreted languages, but simply the diversity and general mess of C build systems. (Of course, some low level programs really do require complex build processes, and those programs are more likely to be written in C than Ruby, but those are the minority.)
And this is something a package manager can significantly, simply by providing less flexibility and therefore more standardization. Ideally, if you can build one package from the package manager, you can build 'em all.
Somewhat belated disclaimer: I haven't given conan.io more than a quick look, so I don't know how well it actually solves any of the preceding, heh. But I've wanted to see a good C package manager for a while, and I fully intend to check this one out when I have the chance.
The primary means to distribute open source software is via source code. Automagical technologies to assemble all the source code for a dependency tree, and/or automagical technologies to fetch precompiled dependency trees with curated options, versions, component granularities, etc are extremely non-trivial problems. To date the only examples of this done with a measure of effectiveness are actually entire operating system distributions backed by software like apt, yum, pkg-src, etc.
Primarily because of the total failure in this regard of windows and osx, individual languages themselves starting advancing ecosystem centric dependency management with files retrievable from servers hosted on line. At best, this style strategy solves some of the problems some of the time, and at worst breaks the ability to make things even work the old way - ie manual download, assembly, and compilation of source code dependency trees.
In the context of largely portable, interpreted code, this sort of thing is at plausible. In the context of native code with multi-dimensional matrices of OS/Compiler/libc/versions/flags/phases of the moon, the notion of a repository of precompiled binaries to even enough cover "common cases" to be thought of as a language level feature.
In the case of source dependency management, exposing the information about where and how things are installed and reachable for use in the build environment of the foreground project is hard enough, and trying to be able to hold cross platform build recipes of some sort is literally the creation of yet another operating system. Take homebrew for example.
Perhaps some sort of unification wrapper over apt and pkg-config is a thought, but honestly can't even guess what problems can be addressed with something like this.
In all seriousness the only cross platform solution maintaining build environments is to just host all builds on linux, use mingw64 and osxcross along with apt or yum + regular pkg-config, git, etc.
This looks like someone who thinks things like pip and gem make sense trying to overlay that paradigm to a broader scope. Or maybe I'm just too old to "get it". Please clue me in?