Why is there a desire to include things like magit in emacs? (Same question regarding e.g. org-mode). Wouldn't it be better to have a lean core, abiding by the strict FSF rules, and a high quality package ecosystem and package manager? It needn't be a barrier to beginners, high quality "distributions" bundling the great packages like magit and org-mode would appear immediately (kind of exist already, like spacemacs).
You tend to have the same kinds of problems as early Linux distros did - namely, fragmentation that accidentally heightens the barrier to entry.
Aside from that, much of Emacs's power comes from the ecosystem of Emacs Lisp applications and libraries that come with the base install. I learned Emacs in the first place because of Tramp's appeal, and shortly thereafter discovered in org-mode an amazingly powerful tool I'd never realized I wanted so badly. Absent either of those, I probably would not have kept using Emacs for long.
Finally, there's the question of support. A few years back, when I was much more heavily involved in helping new Emacs users on sites like Stack Overflow, I found over time that the problems people were having tended more often than not to originate not in Emacs per se, but rather in the (mis)behavior of the various collections of packages that came with one or another of the "starter kits" then popular, and beset by the same fragmentation mentioned above. When you start out not with Emacs, but with Emacs plus a large collection of random packages tied together with customizations and init unique to the distribution, it's much harder to find useful information when something goes wrong, because there probably aren't all that many people using the same thing you are - and, being very new, you likely don't yet know how to have a productive conversation with Emacs itself, or even that such conversations are possible.
Conversely, when you have a problem with something that's part of Emacs, you're much more likely to find something or someone that can help you get past it, because ~everyone using Emacs is running the same code you are, and someone has almost certainly figured out how to solve the same problem before. Fragmenting the ecosystem makes this less likely; incorporating popular packages into Emacs, where that can be accomplished, makes it more so.
As a not very committed and not very expert Emacs user, I find annoying fragmentation in Emacs configuration not at the level of individual packages, but of fundamental mechanisms with inconsistent alternatives.
- defining customization groups and variables for one's package vs. not bothering and using semi-documented raw global variables (many actively despise the customization system)
- properly using the Customization pages vs. just overriding the variables in the init file
-packages that need to be built from a cloned Git (usually Github) repository, packages that want to be "installed" by hand, packages that use the "standard" package manager, alternative package managers, packages offering more than one of the above (or multiple sources with different versions)
- loading and configuring packages with require statements and setting variables; with use-package in self-contained blocks; implicitly or almost implicitly; in ad hoc ways.
Sometimes Emacs itself contributes to the mess; when I thought I exported my faces settings to a theme the "theme" file actually came to contain unrelated settings, preserving obsolete configuration of removed packages and amply overlapping and conflicting with the init file; now, for example, nyan-cat mode only displays a proper wavy trail if I use this magical theme. I'd need to spend hours dismantling the "theme" line by line.
I agree that Emacs seriously suffers from "anti-Python syndrome" (or maybe a better term is just "Lisp syndrome") in that there's many ways to do a given thing and no immediately clear "best" option. However, while it may not be immediately clear, I think almost everything you mention here has a "pseudo-consensus" best option. Not something everyone does, but that most people do, and that you're most likely to see recommended as best practice.
- You should be defining groups and variables to provide better documentation and allow use of the interface. But you shouldn't actually use the interface to do customization (put it in your personal configuration files instead). The interface is both pretty annoying and disallows documentation of why exactly you're setting certain options.
- You use the standard package manager (package.el) and ELPA, MELPA, and maybe the Org repo. There are a few of these which will require external tools of some sort, in which case fair enough, that's an extra step. But that's still pretty standard in my view (although definitely doing that automatically would be great). I don't think I have any packages that require alternative package managers anymore.
- Use use-package. No real question at this point.
I do agree that themes are in general a nightmare.
From my experience as a developer and maintainer of several packages now available via MELPA and MELPA Stable, I can say with confidence that the standards for new package submission are quite high - indeed, my trips through that review process, and the excellent feedback I have on all occasions received, have taught me a great deal about how to build Emacs libraries and packages in a reliable, maintainable, usable fashion.
Unfortunately, there is a great deal of older Emacs Lisp code out there, much of which was not written to any perceptible standard whatsoever beyond "works on my machine lol ¯\_(ツ)_/¯". I say this not to disparage the people who wrote and released that code - indeed, a lot of what's in my own ~/.emacs.d, and which I use daily, comes of the same sort of origin. But either way, the Emacs ecosystem has a serious legacy code problem - something which, of course, you already know full well.
Unfortunately, there's no good single suggestion I can make here.
I can say that learning Emacs Lisp will make it easier to cope with nonsense like people unaccountably refusing to implement package configuration in terms of Customize, and with the occasional package where Customize legitimately isn't flexible enough to support everything that package needs to do.
But learning a whole new programming language, especially one that originates in a paradigm not familiar to most working programmers today, is a hell of a big ask as a prerequisite for effectively using an editor. My own experience is that it's well worth doing so - but if you're already frustrated with the nonsense you've observed, I can hardly blame you for deciding it's not worth your own while.
I can say that, well, it's just a matter of needing to invest the time to learn how to leverage this uniquely powerful tool, and not everything worth doing is easy - but, quite aside from being rude and insulting, such a claim has the same problem as the one I just talked about. I mean, I started using Emacs eight years ago, and it took me three or four of those to achieve something that can reasonably be called proficiency. That's a hell of a learning curve.
I can't reasonably say that the problems you're seeing don't exist, or result from error or misperception on your part rather than failure on the part of the ecosystem. (No one else can reasonably say that, either, and I misdoubt some have tried.) Among other traits, Emacs is older than the entire user interface paradigm that's become ubiquitous since 1985, and that makes many of its core metaphors seem either strange or strangely expressed. "Theme", for example, which everywhere else means a specific visual style packaged as a unit with a name, in Emacs means an entire set of Customize configurations packaged likewise. And then Emacs also, separately, has lately gained themes of the newer sort, with the same name referring to both! Of course it's a huge gotcha for the new user. And there are many more.
All I can say is: the Emacs core maintainers, and those of us building new packages for public release, are working the problem - doing our best to relieve Emacs of the manifold architectural and ecosystemic burdens which make it so easy for people to reasonably regard the platform and its partisans as relics of the past, and replace those legacies with a structure and culture sound enough to support Emacs' next 30 years. There's an almighty lot to do, and as with any free or open source project, never enough capable hands for all the doing. But as we can, we are.
1. Your main point is clearly correct. But the counterpoint is that it's not scaleable for the emacs core developers to take the responsibility of maintaining the code from every third-party package that seems useful enough to include. A few years ago I wrote quite a bit of code for org-mode, so I've seen how this works. The org-mode team prepares a release (err, no test suite though! At least not back then.) And then some weeks/months ahead of a major emacs release, the org-mode codebase at an agreed commit gets copied over into the emacs code base. But from that point on, emacs core devs are on the hook for any mistakes we made. That clearly doesn't scale to many packages. So what you said notwithstanding, the only viable option long term is to make the package management system excellent, and to make the API via which packages interact with emacs core excellent.
2. Yes to what you say about the "starter kits". But things have completely changed since then with the arrival of package management and package repositories.
Essentially the same argument as linux-style OS vs freebsd-style OS.
Its worth pointing out the analogy that most OS have git as an addon and have emacs itself as an addon. Once someone has gone to the effort of installing git, and installing emacs, installing magit isn't a noticeable or significant barrier.
Another interesting thing for non-emacs users to consider is this saves approximately zero time. A quick
grep -c use-package .emacs.d/init.el
shows I import about 36 packages and I'm kinda a minimalist emacs user. The most effort I could possibly save would be about 3%. Its not like I'm going to stop using cider or flycheck if magit is bundled.
require simply loads a feature, assuming it exists. use-package can do a whole lot more, including installing the package, configuring auto-loads, setting up key-bindings, pre- and post-load configuration, etc.
It's a third-party package itself at this point, but I believe it's going to be included in Emacs before too long.
Think Python: One of the biggest strengths of the language was always the "Batteries Included" philosophy, which took Perl's CPAN idea (massively important for Perl's success, BTW) one step further and packaged common libraries into the base install. People got up and running faster, and Python prospered.
OTOH, another big strength is the near universal adoption of certain fast moving third party packages. For example, everyone uses Requests because it's better in almost every way than the built-in stuff, but because it's not included with Python its developer can move at a much faster pace.
That's a hard comparison to make, but I'm sure if the Python developers had a package they thought would be as well-loved as org-mode is among Emacs users, they'd package it in with the base install.
