Kubernetes Machine Learning Hacked for Crypto Mining in Azure Cloud

Hackers managed to exploit user misconfigurations -- apparently done for convenience -- to launch crypto mining campaigns leveraging powerful Kubernetes machine learning nodes in the Azure cloud, Microsoft announced earlier this month.

Specifically, the attackers targeted Kubeflow, a machine learning platform for working with Kubernetes, the popular container orchestration system.

"Kubeflow is an open-source project, started as a project for running TensorFlow jobs on Kubernetes. Kubeflow has grown and become a popular framework for running machine learning tasks in Kubernetes. Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs. This fact makes Kubernetes clusters that are used for ML tasks a perfect target for crypto mining campaigns, which was the aim of this attack," said Yossi Weizman, security research software engineer, Azure Security Center, in a June 10 blog post.

Crypto mining (or cryptocurrency mining or bitcoin mining) is a way to generate digital currency wealth by leveraging powerful computing power. While it's not illegal, it requires tremendous computing effort for usually minimal gains. The whole process is explained here.

Hackers were able to turn Kubeflow machine learning to cryptocurrency mining by taking advantage of user misconfigurations, having to do with dashboards deployed in Kubeflow clusters that expose Kubeflow UI functionality. Weizman summarized this:

The dashboard is exposed by Istio ingress gateway, which is by default accessible only internally. Therefore, users should use port-forward to access the dashboard (which tunnels the traffic via the Kubernetes API server).

In some cases, users modify the setting of the Istio Service to Load-Balancer which exposes the Service (istio-ingressgateway in the namespace istio-system) to the internet. We believe that some users chose to do it for convenience: without this action, accessing to the dashboard requires tunneling through the Kubernetes API server and isn't direct. By exposing the Service to the internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.
The Azure Security Center earlier published a Kubernetes Threat Matrix detailing the major techniques that are relevant to container orchestration security, with a focus on Kubernetes. Shading the matrix to illustrate the recent campaign results in this:
The recent attack depicted in the Kubernetes Threat Matrix
[Click on image for larger view.] The recent attack depicted in the Kubernetes Threat Matrix (source: Microsoft).

Weizman said the attack affected "tens of Kubernetes clusters" but there was no information provided about how much mining was conducted or if attackers managed to do other nefarious deeds via the exposed dashboards.

Although the Azure Security Center has detected similar campaigns against Kubernetes implementations that leverage exposed services to the internet as an access vector, this is the first Kubeflow-specific attack. For example, Weizman in April described a similar exploit of Kubernetes.

User misconfigurations are a prime enabler of such attacks on Azure and other clouds. Notoriously, the Amazon Web Services (AWS) cloud suffered a long string of such attacks beginning a few years ago.

Weizman in his recent post provided guidance on how organizations can check to see if their clusters are impacted and provided advice going forward, warning that they should be aware of security aspects including:

  1. Authentication and access control to the application.

  2. Monitor the public-facing endpoints of the cluster. Make sure that sensitive interfaces are not exposed to the internet in an unsecure method. You can restrict public load balancers in the cluster by using Azure Policy, which now has integration with Gatekeeper.

  3. Regularly monitor the runtime environment. This includes monitoring the running containers, their images, and the processes that they run.

  4. Allow deployments of only trusted images and scan your images for vulnerabilities. The allowed images in the cluster can be restricted by using Azure Policy.

About the Author

David Ramel is an editor and writer for Converge360.

comments powered by Disqus


  • Microsoft's Tools to Fight Solorigate Attack Are Now Open Source

    Microsoft open sourced homegrown tools it used to check its systems for code related to the recent massive breach of supply chains that the company has named Solorigate.

  • Microsoft's Lander on Blazor Desktop: 'I Don't See a Grand Unified App Model in the Future'

    For all of the talk of unifying the disparate ecosystem of Microsoft-centric developer tooling -- using one framework for apps of all types on all platforms -- Blazor Desktop is not the answer. There isn't one.

  • Firm Automates Legacy Web Forms-to-ASP.NET Core Conversions

    Migration technology uses the Angular web framework and Progress Kendo UI user interface elements to convert ASP.NET Web Forms client code to HTML and CSS, with application business logic converted automatically to ASP.NET Core.

  • New TypeScript 4.2 Tweaks Include Project Explainer

    Microsoft shipped TypeScript 4.2 -- the regular quarterly update to the open source programming language that improves JavaScript with static types -- with a host of tweaks including a way to explain why files are included in a project.

Upcoming Events