)
“If your enemy is secure at all points, be prepared for them. If they are in superior strength, evade them. If your opponent is temperamental, seek to irritate him. Pretend to be weak, that they may grow arrogant. If they are taking their ease, give them no rest. If their forces are united, separate them. If sovereign and subject are in accord, put division between them. Attack them where they are unprepared, appear where you are not expected.”
Sun Tzu
The Military Chinese General and Philosopher Sun Tzu (544 BC – 496 BC) wrote these tactics over two thousand years ago, and it is as applicable today as it ever was. Personal and Corporate data is now regularly targeted and traded by unscrupulous actors who use it to undermine Governments, destabilise markets, intimidate or threaten companies and individuals. Your data, and the data you entrust to others is now the most valuable commodity on earth, and those who want to gain unauthorized access to it, will use every means at their disposal to do so. This is the era of Cybercriminals and understanding their methods and means, is the most important factor to avoid becoming their next victim.
Cybersecurity is a complex subject matter for most people, and the levels of sophistication and ingenuity being applied in some cyberattacks, shows that there is an ever-increasing level of co-operation and exchange of information by the criminal underworld. The fact the Dark Web facilitates the exchange of criminal intelligence and many sophisticated tools that were developed for legitimate business or community benefits, are now often exploited by cybercriminals, and redirected against the market they were intended to benefit. This rich mix of traded underworld information and data and the massive wealth that can be obtained from it, means that there will always be a rapidly evolving cybersecurity threat. As I write this article, news of a Florida water treatment plant being hacked with the perpetrators trying to poison the supply by dramatically increasing the sodium hydroxide levels in the water.
The recent security exploit perpetrated on the Cybersecurity software company SolarWinds is a dramatic example of how bad and ugly it can actually get. Their Cybersecurity product is called Orion and is used by over 300,000 of their customers including more than 425 of the US Fortune 500 companies including Microsoft, Cisco, Intel, Deloitte, Maersk, all ten of the top ten US telecommunications companies, all five branches of the US Military, the US Pentagon, State Department, NASA, National Nuclear Security Administration, Postal Service etc… The attackers managed to access systems that SolarWinds uses to assemble software updates to its Orion product, as the company explained in a filing to the US Security and Exchange Commission on 14th December 2020.
The attackers inserted malicious code into otherwise legitimate software updates in what is referred to as a supply-chain attack, since it infects software as it was under assembly. The breach in SolarWinds went undetected for Months and was only discovered when another prominent cybersecurity firm that used the Orion Cybersecurity product, discovered that the Cybersecurity product itself was the cause of the security breach.
In this article we are going to explore how meticulously planned cyberattacks such as the SolarWinds Orion attack could have been detected and exposed in real-time, rather than being detected by one of their customers months later. I have previously written an article that discussed the various cyberattacks types (which included the type of exploit used against SolarWinds) and explain how and why cyber defence solutions are failing us so spectacularly – the article titled “It is time to re-evaluate cyber defence solutions” is available here.
I will explain how one of the most catastrophic cyberattacks of current times could have been easily avoided if a proactive cybersecurity solution like ACSIA was implemented. The product we built is based entirely on Open Source technology, and one of our security modules utilises a technology called Falco. Falco is an open source library, a kernel module for Linux systems that intercepts the traffic between userspace and kernel space, detecting unexpected application and OS behaviour at runtime. ACSIA uses this module to monitor and inspect real-time network and security activity in the Linux kernel – a key function of cybersecurity deterrence.
The SolarWinds Cyberattack involved cybercriminals compromising the server infrastructure of the platforms used by SolarWinds to build/assemble their Orion platform, using that access to produce and distribute to their customers a ‘trojan horse’ version of their software.
This attack was meticulously planned over months, with the cybercriminals taking their time to study the infrastructure, identifying the development platforms containing the Orion source code, before compromising it. The attackers gained full access to the source code in order to become as familiar with it as the Orion developers themselves, before embedding modified portions of code back into the product that enabled the cybercriminals to communicate inbound and outbound with all SolarWinds customers deploying the modified Orion code.
The phased analysis of this attack can be split into following three sequential stages:
- Compromising the server user account hosting the source code in SolarWinds
- Embedding the portion of malicious code into the Orion product (the backdoor)
- Using/exploiting the backdoor exfiltration of data from all customer systems where the compromised Orion code was deployed.
We will now look at how a proactive cybersecurity technology such as ACSIA would capture and respond to each of these three separate attack vectors and eliminate the threat they represent in real-time.
It is worth recalling that many of the SolarWinds end-customers who were affected by this cyberattack are among the most respected and security conscious organisations, all with very large annual security budgets.
Could SolarWinds have prevented their development servers being compromised?
The company has been tight lipped about how the cybercriminals initially gained access to their environment, so they either do not know the answer to this question or are unwilling to disclose how their security was initially breached. On the basis that they are a Cybersecurity company, the most plausible explanation is that a legitimate user account or an automated source code commit user (non-human user, system or application user) account was compromised.
These are two separate attack methods, so we will see how the proactive ACSIA Cybersecurity product would isolate either option, and eliminate it.
The figure 1 below, we see how ACSIA responds to a valid user account being accessed multiple times from different locations that are not within the corporate network or environment of Dectar.
The above notification details a potential account compromise for account username ‘centos’ and includes details on IP address and geo-location of attempted access. Remediation options provided by ACSIA are available by clicking the appropriate action to Kill the connection, Ban the IP address etc…
In figure 2 below, ACSIA alerts us that a system/application user has accessed the development platform using ssh key-pair as authentication method (as opposed to password). It provides the details such as geographical location from where the user accessed, the IP address, the coordinates.
Again, the remediation options provided by ACSIA are available by clicking the appropriate action to Kill the connection, Ban the IP address etc…
These alert responses from ACSIA are sufficient for an organization to have full visibility about who is attempting to access systems and from where. All remediation actions can be performed manually by choosing the desired action. This is how simple it is for an ACSIA user to repel such types of attack in real-time.
Could the malicious code embedded into the source code of Orion have been prevented?
The method used by the cybercriminals to embed malicious source code into Orion was so perfect, that they must have known and understood the SolarWinds environment as well as legitimate employees of the company. This is a clear indication that they had full access to the source code for a very extended period. As stated earlier, the most plausible explanation on how the SolarWinds platforms were initially illegally accessed was by using compromised user or system accounts. Any alternative explanation would have required injecting the entire malicious code directly into the source code platform which would be impossible without having full access to the resident source code.
This means that the cybercriminals were accessing multiple systems across the development platform for extended periods of time without being detected. Whether they used System Accounts, Application or User Accounts is irrelevant, as they traversed, navigated and studied the Orion source code before embedding malicious code into the product.
Using the kernel monitoring module in ACSIA, any data manipulation would have been detected immediately and promptly alerted to security personnel, highlighting every and any permission or access violations being performed by the cybercriminals (of which there would have been many).
Figure 3 below shows exactly how this type of privilege escalation would have been detected in real-time using ACSIA.
The alert and remediation in figure 3 shows how embedded eBPF filters combine with ACSIA’s bespoke algorithms, provide a simple, short but concise ‘system violation’ alert with appropriate remediation options. In the example provided, the username git without sudo privileges is attempting to perform a privileged action. A typical privilege escalation attack looks exactly like this.
Our example shows that the git user is performing a log rotation, which is a legitimate action, we can therefore ignore this particular alert by closing it. Relating this to the SolarWinds Orion hack, from the instant any system’s user is compromised, ACSIA will generate an alert and remediation option like this, every time the compromised user navigates through the system (or systems) and tries to escalate its privileges. It would also expose any actions to inject or commit changes into source code.
What ACSIA is essentially doing here is to intercept the anomalous activity within the kernel and acting on it. The powerful message here is the ability to listen to the system calls, everything user space communicates into kernel space, every single interaction with the core engine of the operating system, there is nothing capable of bypassing such a granular monitoring of activity on a platform. Therefore, it does not matter how malware and rootkit may try to become invisible or how it obfuscates itself in memory and in system binaries, ACSIA will always distinguish these activities by utilizing our kernel level analysis module.
Could data exfiltration via outbound connections have been prevented in Orion?
While SolarWinds may have been a victim of this cyberattack, the objective was to use the exploit of the Orion product to gain access to their 300,000 customers. The SolarWinds customers using this infected Orion codebase were the real victims of this crime, as the malicious code was created to harvest their data and make an outbound connection to deliver the data to the cybercriminals. To do this the cybercriminals created an external web server used as a command and control centre hosted on Godaddy, with a domain name avsvmcloud.com through which all customer data was transferred to them.
Analysing this final part of the hack, it is apparent that for customers data to be exfiltrated it would require outward bound ports to be opened to the command and control website. This type of activity is also easily detected by the kernel monitoring capabilities of ACSIA which detects the outbound connection through ephemeral ports (unprivileged ports that are used to exfiltrate data).
The below screenshot shows a very similar case:
The above screenshot is the event captured by ACSIA where there is an outbound connection and is explained as follows:
- Kernel event by User null: Detect outbound connections to common miner pool ports on server gitlab.acsia.io.
- Indicates that a mechanism within the application itself has made an outbound connection using a common miner pool port and it is interacting with 3rd party services where the data is being exported. The common miner pool ports are ephemeral ports (>1024) that are typically used by miners but also by trojans, rootkits and malware to exfiltrate data. The outbound connection was initiated by the GitLab application, so that is why the user is null, it is a ‘userless’ process. In the case of Orion software where Orion’s own protocol was used to export data this would have been detected and notified instantly.
- Critical Outbound connection to IP/Port flagged by cryptoioc.ch (command=ruby /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/hooks/post-receive port=8080 ip=127.0.0.1
The message details shows that the outbound connection is flagged by cryptoioc.ch and the library that is making that outbound connection is ‘post-receive’ within GitLab. The port used in this case is 8080 indicating the IP that makes this connection is the loopback interface (localhost) of the server itself.
In our example the alert was triggered because our GitLab server made an outbound connection towards another server. This specific event is where GitLab is running its pipeline through a separate system outside GitLab. We use a runner (CI/CD) to do a pipeline for GitLab and that is how such events are detected in real-time whenever and wherever code is being manipulated.
This is how powerful the kernel level analysis and the exfiltration of data like the case of SolarWinds attack could have been instantly exposed using ACSIA.
Conclusion
Let me be slightly critical given the nature of organizations that were affected by SolarWinds attack (i.e. government agencies and institutions). All of them have very expensive suites of cybersecurity products, provided by large cyber vendors, which are then monitored by teams of equally expensive specialist security personnel. So while the end-user is ultimately responsible for maintaining their own security, the industry itself must take responsibility for the poor design of their vertically oriented, reactive security solutions.
Major vendors tend to oversell their cyber solution using terms like “NextGen”, “Machine Learning” or “Artificial Intelligence” as buzzwords to suggest their technical leadership in the market will provide you with a heretofore unrivalled level of security protection. The reality is that they are reactive solutions that are designed to respond to an attack after it has occurred – never with an offensive capability that will help prevent an attack in the first place. This is what is referred to as a cyber Blind-Spot and is what makes them vulnerable to very specific exploits such as the SolarWinds hack.
The reason why these cyber Blind-Spots exist is primarily because most of the major vendor products are designed by developers, data scientists and marketing people but not by cybersecurity architects. If you doubt my conclusion, did you ever wonder why your security product needs a massive database or in many cases a data lake!
If I was to summarise (in no particular order) why so many Cybersecurity products can be easily bypassed, then the following would apply:
- Most cybersecurity products are obsessively focused on remedying a subset of what a security product should perform, a fact that necessitates the deployment of multiple niche security products.
- Most cybersecurity products are reactive only tools, meaning that you are only alerted to security breaches once a platform has already been compromised
- Most cybersecurity products operate at the perimeter with an encrypted network layer which makes packet inspection impossible.
- Many cybersecurity products are rules based and are easily outsmarted by Zero-day attacks and lack of updates
- Virtually all cybersecurity products generate thousands of false alerts leading to omissions and mistakes