“So let me put it this way, I don’t know how old Metador in particular is as a group, but I think that there’s a great deal of wisdom and experience going into the folks involved.”
Dennis Fisher: Okay, so that was my second follow-up question; that line jumped out at me when I read the report, and I think you guys mentioned it in your presentation but you didn’t really dwell on it, in the way that you just described their activities and the way that they didn’t get scared off when your product was deployed, they didn’t run for the hills kind of led me as a complete amateur to think well, maybe it sounds like this is a group of people that may have had some defensive experience too. They may have been on the other side of the ball at some point, and know how defenders think. Or they’ve just learned by watching over the years, I’m not sure. That’s what jumped out to me reading it and listening to you guys talk about it the other day.
Juan Andres Guerrero-Saade: Yeah, there’s definitely experience there. There’s a few different things that – I hate publicizing the techniques that make life hard for us, but I also kind of have to point at them for the sake of being appraised of the actor and sort of understanding how they operate – but there’s things in the way that they handled their infrastructure, the way they segmented the victims, even some of the tooling that goes into the way that MetaMain works. For example, you as an operator on a target that’s been infected with MetaMain, you have the option to deploy Cobalt Strike and Metasploit selectively onto other targets within the same network and other components that you’re trying to infect. That, to some folks, might appear amateurish. To me, it’s like look at how careful you’re being that you would – even though you already have a well-established foothold with an advanced platform on a victim – you’re still willing to avail yourself of totally burnable, commodity off-the-shelf stuff in order to keep expanding your foothold. That kind of thing is very careful, very pragmatic, no hubris kind of thinking, that I think defines this actor… that you go, okay, you’re not just out here showing off, you are very carefully taking each step with great care. Even the measures they took when they saw our product get deployed were fascinating; I mean first of all, they expanded how Mafalda worked, so that to us shows that it’s a platform that’s still in active development. It’s not just something that they bought and have been using the way that it is. They added something like I want to say 14 additional commands and then wrapped the whole thing in some of the most complicated custom obfuscation we’d ever seen. So God bless Alex from our team, who beat his head against this awful obfuscation and I think his brain was about to melt. But, you have a series of opaque predicates and control flow obfuscation and string encryption and a bunch of other measures that just make it very very very hard to reverse engineer, dynamically, statically, however you want, just very hard to reverse engineer. And then once you get under that wrapping what we realize is they’ve added a bunch of commands and capabilities for the backdoor in order to do things like, what they call non-naive execution which is ways to do the things that they were going to do without involving the normal APIs and aspects of the operating system that are what generic EDR solutions would hook. It’s what the AV normally hooks. So to them it was like look hey, okay, there’s a new contender here. We’re just going to reengineer the platform in a way that avoids what we expect them to be using. It also tells us that they don’t have our product to test against, because even though they’ve done all that, it lit up like a Christmas tree on our console, so we were like okay well nice try. Thanks for the new platform. It also lets us know that you’re not sitting on the product trying test against.
Dennis Fisher: Okay, so if they had a given product to test against they would understand how it’s going to react once it’s deployed, what it’s going to be looking for, what it’s going to hook and then what kind of behaviors they could use to get around that, right?
Juan Andres Guerrero-Saade: I mean that’s very common. VT, VirusTotal is something we all know and love and it’s definitely a defensive measure. There are not just sort of black market versions but also custom in-house versions of those AV farms, what you might call them, where what you’re doing is that. Let’s run it against 60 engines and let’s see what antivirus detects our stuff and from the ones that did, well what exactly set them off? So you go back even to the days of of Flame back in 2012 and Flame had a list of AVs and what it would do as well, if we see McAfee then we’re going to change our file extension to be this other file extension and they don’t check that stuff, and if we see Symantec or Kaspersky or whomever; like they had studied what these different AVs did, what their weaknesses were, and they would modify the platform in situ in order to abuse that. It’s a lot harder these days, in the age of cloud-enabled detection, to do that… you have no idea what the machine learning and sort of cloud side of it and correlating between different endpoints and stuff like that, all that stuff happens on the cloud. And that’s more of a hold your breath and hope area.
Dennis Fisher: Yeah, so in the initial victim, where you guys found this, were you eventually able to eject them from the environment at some point as far as you know?
Juan Andres Guerrero-Saade: Yeah, as far as we know, yes. That’s where you pit the threat intel researcher versus the defender in me, because yes, we evicted them to the extent of our abilities, in collaboration with the customer and that’s where it gets you right, it’s in collaboration with the customer. I’m not going to malign anybody but there are some customers who’ll tell you, hey thank you, we’re busy, goodbye. They don’t care. I hate to say it. But there are some customers who are like that. In this case, I’m conflicted because we definitely evicted them from the Windows side and then we know that there were components in the Linux side of the house in what represents the core network of a telco and that’s where the cool stuff happens. And it took some folks a little too long to get the product onto the Linux servers and we missed all of those Linux… So we saw them communicating. We saw that there were parts of the Windows components in Mafalda that are clearly stealing stuff from Linux implants. But we have no idea what the Linux implants were doing, what they look like, we didn’t get our hands on samples. And it kind of sucks because with telcos and ISPs that’s where the cool stuff is happening. When, I want to say Mandiant did this great research into MessageTap, I think it was called, it was a Linux implant for telcos, I believe used by the Chinese, and because they got their hands on the sample, you could see hey they really care about text messages from this list of phone numbers or from people in this particular region and we’d of course love to know that. But on our side, we see a similar set of components, we have no clue what they were after at that point.
Dennis Fisher: Yeah, and now that this stuff is public I’m sure the teams like this pay attention to – I mean obviously they knew they got got in some way or another, because they adapted to when your product was deployed – so they knew they were at least found in that environment. So you would expect them to adapt their tools and techniques in other and or upcoming intrusions as well, right?
Juan Andres Guerrero-Saade: Absolutely, again a moment where me as a defender versus me as a researcher is kind of caught in a bind, because I want to be able to keep track of these guys. And in some cases, I mean there’s some threat actors who will get very clever and pour a lot of resources into trying to engineer against your defensive product. There’s others that will say hey if S1 is there, don’t deploy and I mean in a sense that’s a qualified victory for customers. But for me, it’s like well there goes a whole white whale that I’m not going to deal with for a significant amount of time. And from wanting to understand the threat landscape, wanting to have real situational awareness, that doesn’t necessarily feel like a victory to me.
Dennis Fisher: Yeah I completely understand that because you had this window into the behavior of an animal that you’d never seen before, you got to observe it, you got to see how it behaved for a certain amount of time and then the window closes, and you’re just like, what’s it doing behind that window? Now what, what are we missing? What else could we have learned?
Juan Andres Guerrero-Saade: That was a big point, I mean honestly, that was a big point in how we structured our release. You mentioned the 30-something-page PDF, there’s another 30 some pages of living technical analysis on a GDoc. We tried this different approach of saying look, here’s all of our reversing notes, and whatever we find, if people share more stuff, if they’re cool with it, we’ll add it to this living document. But part of that was to say, A, LABScon was meant to be something to enable collaboration and the talks we put out from our team, Tom Hegel, from me, Amitai, and Alex and so on, they really were meant to foster collaboration and we tried to do our best to be like look here’s the kimono open, like Tom did his talk was off-the-record even though he published a report but it was an off-the-record version where he could be like look, these are some of the guys we found, this is one of the companies we found, this is one of the tracking methods we’re using, and instead of being coy or just kind of like patting ourselves on the back with a big release, it was more to say look, we don’t want to burn this method, but all of you that are working on this stuff, you should know how we’re doing this thing. With Metador there’s this reminder that even though for the past 10 or so years, there’ve been a lot of amazing discoveries and unbelievable findings from seeing the NSA/CIA, seeing Five Eyes, seeing the Russians and a variety of their teams, the Chinese and a variety of their teams, not just the low-end but some very high-end organizations, Singapore, all these other really amazing countries doing really cool things; Most of those findings are snapshots in time. They give us a really amazing sense of look at how they do things right at that sliver, maybe look at how they did things for the past however many years. But they seldom turn into consistent situational awareness of what these actors are doing. And I think that’s an important reminder and a humbling reminder that we really haven’t bested any of these folks. You can go like gotcha for five seconds, but from a defensive standpoint that doesn’t suddenly turn into we know everything that this Russian team is going to do indefinitely. And it’s a reminder of the grappling and tussling that comes not just with threat intel teams, but also the software and hardware that’s generating our telemetry, and I think that a lot of times we lose the battle on the software development end. Like there is a race to the bottom when it comes to costs, when it comes to being pragmatic, when it comes to not wanting to consume too much CPU resources, as if Chrome and Slack weren’t already doing that on their own. You know there’s all these arguments for why you need to have as little a footprint as possible and the counter-argument as well is look at what these people are doing in firmware and memory and well beyond the pale of what some light logging is going to provide you and how comfortable are you with that.
