the first commenter was right on the env thing helping with ci/cd secret isolation
the open issue is that GHA env is vulnerable to typical npm/pip/conda/etc supply chain attacks. it lets them do funny network requests (both data exfil + integrity, and even quota wastage / DoS):
- GitHub hosted runners should allow inbound/outbound domain safelists, full isolation w internal repo safelisting, or something else like it
- and/or self-hosted runners should have an Azure serverless mode where the host owner can do their own presets. less good b/c trickier to maintain typical safelists
currently , GHA is default-allow for supply chain attacks =\
in theory folks can try docker-in-docker to limit network access, or the equiv lower, and layer on internal repos to prevent 0-days, but this gets gnarly quickly and more sanely handled by the managed CI env we are all paying for
Thanks for taking the time to share your experience. I work on GitHub Actions and am familiar with the limitations you're experiencing. Below is more info on where we're at with each of these issues, using the numbers in your Table of Contents. Also, we have a lot of other new things planned for Actions as you can see in our roadmap. https://github.com/github/roadmap/projects/1?card_filter_que...
2.1 We're starting work on this in the next couple of months. We plan to ship it in early 2022.
2.2 We want to speed up the pace of GitHub Enterprise Server releases, but I don't have more info to share.
2.3 We're looking at ways to not require a GHEC account or "unified" license.
2.4 The limits are much higher with the GitHub hosted runners, but this is a current limit of self-hosted runners.
3.1 It's on our backlog. No date to share.
3.2 I haven't heard this before and thanks for sharing the scenario. We'll think about it.
3.3 This will ship in October.
3.4 We're doing some performance optimizations for GHES 3.4 that should fix this.
3.6 We have a couple API improvements coming later this year.
3.7 We're looking into this, but no dates to share.
We're dedicated to making Actions a great experience. As you would assume, I'm very excited about the future of Actions and getting feedback like this helps us make it better.
3.5 I was excited to see action composition until I realized that the composed action will still be showed as one step and that it can just compose actions instead of whole workflows.
I want to share whole jobs across my repos, including matrixes and all. I could wrap some common workflows into one action, but then the whole complexity will be hidden in a single step.
So I resorted to creating GHA Templates and manually synching them with a CLI tool[1], which isn't great, but at least I don't have to copy-paste changes across repos and I can keep a "main version" of the workflow
Related to 3.3 (You can’t restart a single job of a workflow), is there any reason why the numeric id of a specific job run isn't available anywhere? There's GITHUB_RUN_ID, but that only identifies the workflow run, not the individual jobs, and this isn't unique across run/job restarts. Services like https://coveralls.io/ need an actually unique job run id, and they could also use the numeric id to link to the specific job run in a build matrix.
No, there's no reason why that isn't available. This is a very sensible request and we'll add this information to the context available to workflow runs.
I love GitHub Actions, so much so I’ve published lots of Actions on the Marketplace. I love that I can contribute to better pipelines for all.
That said, I have frustrations with the management of the GitHub Actions repositories: which is to say, there isn’t any. Most issues and Pull Requests languish, without acknowledgement or engagement.
I appreciate that managing issues and contributions is a full time job, and your team probably doesn’t have capacity right now: if you could magic up some capacity, that would be great, but… a more realistic request would be to evaluate capacity and follow Terraforms lead in being open about pausing contributions —- rather than READMEs that encourage contributions, which (despite the best of intentions) will never be reviewed.
Regarding 3.5 (actions calling other actions) – please continue investing in composite actions. They are a well-balanced abstraction for managing workflow complexity. I can see how they could become too powerful and encourage bad patterns (YAML programming), but that's a good problem to have.
We have recently switched to GitHub actions and in addition to the above, there are two others which impact us:
Sometimes the checkout action fetches the wrong commit! The SHA in the GitHub environment variable says one commit, but the actual code is different(!). Because we don't know why this happens we basically need to do an unshallow fetch of the whole repo to be sure we have what we expect.
Using autoscaling self hosted runners, it is not currently possible to instruct the agent to finalise the current job but to accept none after it. This is essential to avoid broken workflows while scaling in. Gitlab supports this via a signal, but there is no equivalent.
> Sometimes the checkout action fetches the wrong commit!
I'm a product manager at GitHub, and I'm investigating this now. Feel free to drop me an email (ethomson@github.com) if you can share more information about the workflow runs that were affected by this.
> Using autoscaling self hosted runners, it is not currently possible to instruct the agent to finalise the current job but to accept none after it.
We're building this functionality out right now, so that you can have self-hosted runners that will run a single job and we will let you subscribe to webhooks that will let you scale up/down your fleet of runners.
From an outside perspective it looks like neither of them is going away anytime soon and that they just want you to use the one that you feel more comfortable with.
It's pretty clear that Azure DevOps is going the way of the dodo. They haven't even finished migrating their "old" pipelines to the "new" pipelines, and I'm already seeing some things that are GitHub Actions only.
They never officially killed Silverlight so I doubt they'll kill Azure DevOps either. They most likely will at some point deemphasize Azure DevOps to favor GitHub Actions tho.
I wouldn't be surprised if Azure DevOps still pops up in random places for decades to come or if they decide to "unify" the two products without changing the underlying codebases (like OneDrive for business actually using Sharepoint under the hood).
Unlike Google, Microsoft seems to commit to legacy software to a fault. As long as there are enterprises paying good money for support, it'll probably not go away completely.
EDIT: Microsoft only stopped licensing OEM Windows for Workgroups 3.11 in 2008 (15 years after the first release of WfW 3.11). Apparently it was still being used by some embedded device manufacturers: https://docs.microsoft.com/en-us/archive/blogs/jcoyne/its-th...
You don’t need to sign up. It’s open to everyone with a GitHub Account. Also, you don’t need to use Actions. You can just push a container. Here’s a doc that shows you how to push a container. https://docs.github.com/en/packages/getting-started-with-git...
I’m a product manager on Azure Pipelines. Can you tell me more about what’s missing in the docs and language support to make it better. Email works as well jepling at microsoft dot com.
Open source projects get unlimited self-hosted agents. Let me know if you aren't experiencing that. I'll make sure it's clear in the docs and the website. jepling at microsoft dot com.
I'm a product manager on Azure Pipelines. I'd like to learn more about your pipeline so we can improve the performance of our hosted build agents. It's something we're improving right now. Feel free to email me. jepling at microsoft dot com
Congratulations on the launch, and for your support of open source projects! I'm interested in extending our bulk builds of the OCaml language community packages (https://opam.ocaml.org) to Windows. Do Azure Pipelines support Windows Docker containers yet? Having somewhere to robustly build native Windows images is quite difficult at the moment.
One of the other things on my wishlist is for some sort of standard for all these CI/CD pipeline agents. So far, on our build cluster we have BuildKite, GitLab, Azure Pipelines, <insert CI provider> agents, each of which have slightly different platform support. It's particularly challenging when trying to extend our CI to slightly less popular operating systems such as OpenBSD or exotic or newer architectures such as ppc64le or RISC-V. It wasn't clear from the Azure documentation whether or not the agent is open source or not.
Today, we use a combination of BuildKite and GitLab to coordinate the multi-OS, multi-CPU-arch Docker images that then get pushed to the Hub. The first provider to give us a one-stop coordinated multiarch build pipeline will save open source projects like OCaml so much trouble for testing and publishing our releases!
We absolutely support building Windows Docker Images. At the moment you are limited to either Server 2016 or Server 1803 based images but that is something we are working on enhancing. I would be interested in understanding your needs at a deeper level. Please ping me on twitter @chrisrpatterson if you are interested.
[Dave M - Azure DevOps PM] Also includes more free: 1,800 monthly minutes for private projects instead of 240 and 10 free concurrent jobs for CI/CD. GitHub marketplace extension for easier config from GitHub. But nearly the same functionally as Chris indicated.
Azure Pipelines has all of the same features on VSTS Build and Release but adds an option for defining your CI pipeline in YAML (this feature was previously in preview).
I'm currently reevaluating the CI/CD story for my company. We're a small startup with ~20 engineers. I would like to give you some feedback and questions on Azure Pipelines.
We're currently using gitlab.com for code hosting and CI/CD. We're in the Silver / Premium pricing tier so have most of the premium features. Our software stack is mostly on GKE Clusters on GCP and we also run gitlab runner in a GKE cluster.
We're honestly not super happy with gitlab.com due to performance, stability and UX issues - overall not super happy with the overall quality/depth of the product. That's why we're considering to make the switch to github.com and reconsider the whole CI/CD story. I evaluated about 20 CI/CD platforms, but unfortunately didn't find the perfect one. There was always at least one important feature missing or the management of the system (i.e. jenkins, concourse) would've been too much overhead for a company of our size.
My preliminary conclusion is that we should probably stay with GitLab CI/CD for now (even when switching to github.com for code hosting) since it is good on most dimensions (far from perfect though) and has low management overhead.
However I gave yesterday Azure Pipelines also a shot and liked many things I saw.
My impressions were like that:
Pros:
- Flexible build system / environment, can run on VMs, containers, self-hosted agents etc.
- Reasonably complex builds can be expressed in the yaml.
- Reasonable pricing
- Big plus: Very sophisticated release / deployment pipeline management. Nice sweet spot between solutions with no / limited CD support (e.g. circleci, gitlab) and complex systems like spinnaker.
- very good github integration (uses checks API)
Cons:
- Big con: Self-hosted agents cannot autoscale. For comparison gitlab runner k8s executor spawns kubernetes pods on demand so the infra / cluster footprint is really small in idle times during the weekend or nights. In comparison azure pipelines seems to only work with non-ephemeral agents and one has to set a fixed statefulset count: https://github.com/Azure/helm-vsts-agent
- Big con: Release Pipelines are only configurable via UI instead of code
- No explicit docker layer caching support (vs codefresh, circleci)
- Many out-of-box integrations / features are tied to Azure Cloud, for other clouds (i.e. GCP) one has to do write custom scripts or integrations.
The two biggest cons for me are really the non-autoscaling self-hosted agents and that release pipelines are only configurable via the UI. The release pipelines only-UI problem I consider a blocker for us.
In other words: If you would allow configuring release pipelines in code we would probably switch to github.com + Azure pipelines. FYI I prefer expressing complex pipelines in a real and common programming language like TypeScript or Python3 (think Jenkins: Groovy, Airflow: Python, TeamCity: Kotlin Script etc.) over YAML, even though I can live with YAML for now.
Is your team working on enhancements to any of that and what would be the rough timeline?
Thanks for the detailed and thoughtful feedback. I'd love to get on a call with you to discuss your needs and some of the items on our roadmap. If you're interested, email me at jepling@microsoft.com.
Microsoft is moving to Git and we use Team Services / TFS as our Git server for all private repositories. GitHub is only used for OSS since that's where the OSS community is.
It's also not just the community, but GitHub provides significantly better integration support, than GitLab. Since GitHub has such a robust API, it's easier to create bots and what not, to help better manage large open source projects.
