Warren Kruze: Hello, and thank you for joining our panel presentation on the IoT. We have a great panel here for you today. I’m really looking forward to this. This is a panel that we’ve been working on and the content that we’ve been working on specifically for the Magnet Virtual Summit 21. Thank you all for joining.
Well quickly, you have the bios on the webpage, so we’ll skip the bios, but we’ll go around the room. And the virtual room, hopefully next year at the next Magnet we’ll be in a real room. Like in the past. But we’ll go around the virtual room and let the panelists introduce themselves quickly. My name is Warren Cruz. I’m a vice president with Concilio, a global electronic discovery service provider. Ken?
Ken Oliver: Great, thanks Warren. Good afternoon. My name is Ken Oliver. I’m a senior director at Concilio as well.
Norman Rankis: Good afternoon. My name is Norman Rankis and I’m a consultant and I’ve been in the business for 40 years.
Patrick Bland: Patrick Bland here. I’m the CTO of Splunk technology looking forward to talking about it with you guys.
Trey Amick: Awesome. And I’m Trey Amick, one of the directors here at Magnet Forensics for the forensic consultant team. Just want to thank, you know, Patrick, Norman, Ken, Warren, all of you all for being a part of MVS 2021. And Warren, thanks for having me on y’all’s panel today. Looking forward to it.
Warren Kruze: Absolutely. Thanks. Yeah. Let me echo that. Thank you all for the time putting this together. I really think it’s pretty good. And I think you’ll, we’re gonna start off really quickly with what is IoT, so that everyone is on the same page. I still, you know, I’ve been tracking IoT for as long as I can remember. But I still find out things and, you know, Norman and I were talking the other day and I’ve found out new things that he’s going to talk about. So just so everyone’s on the same page, I’m going to ask Ken to take us through a little high level. What is IoT?
Ken Oliver: Sure. Thanks. Thanks again, Warren. Again, the internet of things is generally described as a network of physical objects that are embedded with sensors software and other technologies for the purpose of connecting and exchanging data with other devices and specimens over the internet. But again, the short answer is really anything that’s connected to the internet. And as you can see by some of these examples, you know, it really does range from anything from, you know, home equipment, all the way through enterprise and you know manufacturing devices and cars, et cetera.
And when we talk about the internet of things, we want to talk about the growth. And as we see things progressing over the next several years, you know, 55% of all data is forecasted to be generated by the internet of things by 2025. And again, as you can see by this chart, you can go forward, Warren. And as you can see by this chart, again, just some examples of the numbers of devices that are out in the wild globally. You know, it’s just some astounding facts, you know, 50.1 billion door locks that are automated and have some type of inner connection to the internet or other systems. I mean, it’s really interesting to see how this has grown in the last 10 to 20 years.
Warren Kruze: It’s kind of amazing. You look at how fast that’s going up and, you know, we’re at 50.1 billion at the top now, and I’m sure, you know, it’s even gone from there as well. We’re going to be talking about some of the uses of IOT, some of the ways to collect it, some ways that it’s being collected that you might not even know about. And that’s one of the things that worries me, you know, as someone that does investigations, you show up, and not to say that anyone on the Magnet Virtual Conference would do this, but I envision, you know, at some point someone shipping me a smart thermostat and saying, Hey, can you do forensics on this smart thermostat? Obviously we hope that doesn’t happen, but there’s so much data out there. And you know, kind of trying to keep track of it boggles my mind.
And, you know, I have these what I call aha moments. And this was one of my aha moments, when I actually had more hair, which, you know, I need to edit this picture a little, take some of the hair out. But I was studying computer forensics, doing a master’s, I actually have my masters and my undergrad from Champlain. And I was on campus in their forensic lab, which if you ever have a chance to visit it’s a really cool place, they do a lot of, you know, cutting edge research. And at the time this was, I should know this, when did I get my degree? I don’t know. I can’t read my diploma at this time, but a few years ago when, you know, the Google glasses were just coming out they were doing research on it and, you know, I had this aha moment and I started tracking IOT back then saying, Hey, you know, we might need to collect this data at some point.
My recent aha moment or, you know, the light bulb went off, was two occasions. On the left there is a screen capture of my cell phone. And I got an alert from my security system saying the kitchen door was closed. And I was like, well, that’s kind of weird because we weren’t home. And I have some cameras around the house. So I went to the videos thinking, you know, why is my door showing as closing when we weren’t home at the time? And I noticed that I only had alerts being saved. I didn’t know that. I’d figured I’d be able to watch video going back, but, you know, it turns out you had to pay more for all the video to be stored. And that’s when I had an aha moment saying, wow, you know, I wish I knew that before the alarm system said the door was closed. You know, why was it open? I don’t know, because there was no alert.
On the right hand of this screen was my other aha moment. I got a smart… and this is, you know, everything is smart, everything is IOT these days. I got a smart humidor for my cigars. I like an occasional cigar. And even though my humidor has a glass front and I could literally look in and see what the temperature is, being a geek, I had to have this thing connected to the internet, so I could like look at my cell phone anywhere in the world and know, what my humidity and what my temperature is. But what I noticed was the device, the picture in the middle there, the device itself only keeps very limited information, but when you log in and you’re taken to the cloud, you could go back in time.
And that was my second aha moment saying, oh man, again, you know, what if this was some cigar superstore that had millions of dollars in inventory and someone got in and, you know, either raised the humidity or lowered the humidity or the temperature and ruined millions of dollars’ worth of stock. And someone sends me this humidor, you know, you only have a little picture here. The rest of the data is somewhere else. And that’s what we’re going to spend some time talking about. Where else might the data be? Ken?
Patrick Bland: Ken, I think you’re on mute.
Ken Oliver: Okay. Of course. Sorry about that. Anyway. Thanks Warren. And thanks for giving us a couple of examples, but again, here, we just wanted to, you know, for consumer or smart home products, we wanted to kind of cover the array of what’s out there. And as you can see again, you know, there’s several different examples from, you know, refrigerators and thermostats all the way through, you know, outlets and door locks and home security. So again, this is just an understanding of showing how and where data may reside in typical homes.
Warren Kruze: In typical homes, but when we’re doing cases, there’s also a lot of companies. Ken’s earlier slide showed, you know, how many enterprises are using IOT. And I wanted Norman to talk about, he’s a consultant that works with industrial IOT, and I wanted Norman to talk a little bit about what he sees in industrial IOT.
Norman Rankis: Well, industrial is at a boom right now, and I really need to stress that the seminar is for your benefit for you to really benefit with an understanding of how IOT or IIOT, industrial internet of things, is rapidly moving at lightspeeds. Within a manufacturing environment, you’ll see from the screen that almost every aspect of a business process is being monitored through IOT devices. The question is why, and the answer is to reduce costs. You can [indecipherable] start reducing the, you know man hours and counting electrical kilowatts at meters. You can now do that with IOT devices. So the savings is tremendous. And when you look at a manufacturing world, the main goal is just not quality of product, but to do it at a very efficient cost. And so that’s what the IOT is doing for manufacturing.
And so again, I need to stress that every leader is a teacher and every teacher is a leader. So I’m asking the participants of this seminar to take this information as educational so that it can improve your process when you’re doing forensics investigations. So if we can move on to the next slide.
Warren Kruze: And disclaimer that I should have, I apologize, I should have started off with we’re all here, you know, on our own, talking about information not specific to any company or any, you know, specific case. This is just for educational purposes, not our views of our companies, et cetera.
Norman Rankis: So again, with the smart factory, and, you know, I read an article about a factory in Japan that has probably about 95% of that factory running under robotics and IOT devices. And so that basically is what my charge is when I’ve worked with manufacturing companies, is to bring in a process that in which we can automate and use industrial internet of things.
If we could change the screen, please. So how I categorize IOT is very similar to a grammar school. There are many types of IOT devices out there. Some what I call third graders, some are, you know, 10th and 11th, 12th graders right to the point we’ll have in a manufacturing environment, a device that does everything that it itself is a computer. It will transmit certificates and of course the digital data from the sensors that it has, but also there’s what we call programmable logic controllers, which are PLCs. There were manufacturing companies such as Rockwell international Siemens, and the list goes on that had been producing PLCs for years. The key thing is that PLCs can be very expensive. I’ve seen them range from $3,000 per a PLC all the way up to $200,000 a PLC networks. Again, those PLCs, I’m going to show you for going to the next slide.
They are, they are wired. So what we’re looking here is we’re looking at a NEMA box and within that NEMA box, you’ll see the wiring to the PLCs. Those PLCs are actual computers themselves. This, in this case, these are Rockwell computer PLCs and on the right-hand side, you’ll see we have a wireless sensor and that sensor could pick up on levels of fluid. It could pick up on temperatures or any sort of variant that you can think of. And also the wireless PLCs they’ll be connected to a hard wire also through the network to a NEMA box, which is hooked up to the PLC. So in the manufacturing environment, this is you know, very, very key, very important. And it also leads to issues. As I pointed out that PLCs are, you know, can be quite expensive and we also pointed out that manufacturing’s goal is to keep the products of quality, but also to reduce the cost.
So PLCs have to be upgraded just like Windows, Macs, et cetera. And so it can be very costly to upgrade your PLC with the latest firmware. And unfortunately there are PLCs out there and there’s quite a few that are at end of life and have been end of life. And so for a plant to do a capital budget of multimillion dollars to exchange its PLCs, the upgraded ones, is a tremendous investment and also could be a hardship on the cost of their products. So if we could move on forward to the next slide.
So here we see a, basically a PLC network in this case, we’re using a water sensor and that water sensor can measure, you know, gasoline, could actually measure just water, any type of liquid. They exist in our battleships aircraft carriers and that PLC is hooked up to a computer or in turn that computer is what we call an HDMI. All right, it’s a human digital machine interface. And you’ll see from the screen, it denotes what it’s actually functioning. And in this case, it’s looking at valves and giving you a reading of data, those valves.
Now on a side note, we just recently had heard in the news and looked at the attack of the sewage water, sewage plan out in Florida. That was, they state, was done remotely through a what we call, you know, a teams process. Could be a Bomgar remote software that’s used internally. It was also cited that the machine was an outdated Windows 7.
So all those cases, you know, show that you need to have your network totally up to date with the latest patches. I think what’s important is that you realize that this data, as Warren was pointing out, there’s a lot of data. I’m working at one plant that we capture 5,000 IoT type tags, every 100 milliseconds. So you can understand the, you know, terabytes of data that can exist that allow engineers to be able to say, well, this machine is not running well, but it ran great four years ago, let’s go look at what the settings of that machine was for four years. And so using what we call historians, which are basically high-speed SQL databases, we can pull that data and create queries and trends. I worked on a situation where we had an explosion and we were able to pull the SQL data to map it just right before the explosion to show whether the operator changed some values of that machine.
So, you know, one case, you know, water may not do so well with hot molten iron. And that can cause some issues, I think, as you can understand. So in the manufacturing environment, basically the PLCs, IoTs are now starting to really intelligently run the plant. There are also issues with IoTs. If you have a network that, for example, you have somebody that hits a switch and kills the power. You’re going to lose those, the switch, and it comes back up and you’re having you know, poor data that can cause issues to how your machines are running or could put you know, could create a safety issue for your workers. So, you know, this is a very, you know, delicate environment. It’s monitored constantly with most manufacturing plants. And what I’ll be doing is showing you that, you know, besides the water sewage case in Florida, that hackers can get into PLCs just as you get into computers, because PLCs are assigned an IP address and PLCs they do store their data.
And so let’s move on to the next slide so we can take a, you know, closer look. So what I’m stressing here is that, you know, cyber attacks in manufacturings is getting you know, higher in volume. And it’s extremely important that manufacturing organizations, companies keep their software, their hardware at the latest you know, possible, which includes switches. And switches, I’ve seen manufacturing plants putting in the wrong switch. For example, there are what I call a backend office, which is like the Ciscos. And you can look at those in your office and they have nice air vents and so forth, but that may be the type of switch that can cause a horrible accident within the plant. Because if you have a gas leak and that switch is not a sealed switch and again, there’s a lot of manufacturers that produce sealed network switches, you’re going to have an explosion and I’ve seen that happen. So, you know, it’s very important that when you’re doing forensics, that you have a total understanding of that architecture of that network and how it’s laid out and designed and what type of equipment they’re using. So that’s, that’s key.
So if you can hit on the now thing. Okay, good. So what I wanted to show you is that this is with the software program, that’s running a machine. So we’re looking at the PLC logic. And there’s several types of logic. This is what we’re looking at, ladder logic and this is widely used. The electrician engineer most likely would like, prefer to prefer this and you’ll see that the code are within those boxes. And so there are so many tools like WW client that once you have the IP address of that PLC, that you can you know, get into the PLC and make a change if, I don’t know if you can see that 2000, so that may be you know, 2000 percentile of oxygen to gas. So if I wanted to be a bad hacker I could turn that up to a hundred percent and start creating an unsafe situation of a potential explosion.
So not that you know, I want to play this lightly. Again, this is educational that, you know, once, you know, PLCs are running off of IP numbers and those PLCs along with IoTs you know, again, that running your operation, you know, need to be secured in the ability to get to that PLC. And so manufacturing companies are using quite a few.
Now, what I’m showing here with this screen is a on the internet. I don’t know if you use Shodan, it’s S- H -O -D- A- N.com. And all you need to do is put in IoT or cameras or VSAT. And what Shodan does, it sends out spiders to all the IPs out there that exist, and it looks for open ports. So this is widely on the internet. It’s scary because, you know, putting a VSAT, you know, slash 8080 to get into a certain port you can get into cameras and what’s scary about it is you can get into ships throughout and see, and actually get into their IoT devices if they’re not secure. And this is why I’ve got to stress on an educational level that again, you’ve got to have your software, your hardware at any type of circumference defense systems that you can afford to put in. And I’m saying, it’s basically, you cannot affordto be lackadaisical about this. And you’ll be shocked as to the number of, you know, hundreds of companies that are wide open on the internet. And this is just one of the many sites that hackers will use to get into IIoT.
Warren Kruse: Awesome. Thanks. Thank you Norman. I, every time I spoke with him in the past and, you know, he talks about explosions you know, it just takes this whole IoT, in my opinion, to a different level. You know, again, I’m looking at it from what I have in my home or what I have in my office or company, but then when you talk about its use in industrial and how changing something could have devastating effects, really should make us all think about this a little bit more seriously. Ken tell us where the data’s stored, because it seems to be all over the place. Ken you’re muted again.
Ken Oliver: I got it. I got it this time, thanks Warren.
Warren Kruze: There you go. It’s only been a year of doing Zooms Ken.
Ken Oliver: Now, it really can. Yeah, it really can be stored in many different places. And as you mentioned with the humidor example, you know, you can find some data on local devices. And then, you know, there may, data may be resident in the cloud, maybe in a database application. It may be on a server somewhere, locally in an enterprise. It really depends. And one of the things that you really want to you know, be sensitive to is, again, the data may be you know, it may be specific to time period. So it may only log data for certain periods of time, 24 hours, 48 hours a year. And just because you have the physical device doesn’t necessarily mean you have the data. So again, as you mentioned, Warren, with the humidor, you know, you may, if you get the actual physical device, you may get, you know, a certain portion of that, but the resident, the data may reside, you know, wholly over a longer period in the cloud or in a database somewhere.
Warren Kruse: I wanted to see if anyone in the panel had any comments on possession, custody and control before Patrick, I asked Patrick to join when he was doing some stuff, using a tool that he’ll describe. Taking the raw IoT data and making it usable and exportable. But it, again, another aha moment that I had, you know, you think the IoT data might have it, or, you know, it might be at some edge computer or in the cloud, or in a tool that Patrick’s going to talk about those types of tools that, you know, that collect the data and, you know, make it readable. Anyone else have any other cautionary tales, or information on possession, custody and control?
Ken Oliver: I think there’s just some good points on here. Just, you know, really have a clear understanding of what the devices are. And you know, sometimes you may have to do some research to have an understanding of whether or not you have to power the device, if there’s an issue with powering the device on and off, or disconnecting it from, you know, the internet or its interconnected devices. So there’s just some sensitivities you have to think about when you know, considering how to collect the data and whether it’s volatile or not.
Patrick Bland: And Warren, you were making a point that, just previously, that the data might not be available for a long time. You know, one thing to consider too, as with a lot of this IoT based data, it’s, it’s more metric driven, right? Like what Norman was talking about, monitoring the health of the different industrial assets that might be on a manufacturing floor or, you know, take for instance, like a telecom provider that might be monitoring, you know, information coming off, a set top boxes, right? The value might not be in the actual events themselves individually. But the value that they’re getting out of that data might be in larger amounts of time. Right and so there’s tools out there, right? They can do analytics of data flowing over the stream, right, without it actually being persisted to this. So the raw events that we’re talking about might not ever get preserved anywhere.
Right and so these are additional things just to think about in terms of, you know, from a preservation standpoint and where you’re actually looking for data. You know, as I’ll talk about here in a few seconds, like getting data from all these different systems, from all these different devices, there might be five or ten ways that data is being streamed into some kind of tool or sometimes a storage repository.
Warren Kruse: And just real quick before I turn it back to Patrick you know, this was an example that our friends at Ryan in Dallas shared with me that I thought was pretty cool. Again, data that’s being collected for not investigative use, but you could take all these different data sources like tolls and GPS data and put it together and then figure out, you know, where the person was, what were they doing, you know, where were they so on and so forth. And it’s pretty cool when you have these analytics tools that could, you know, form the companies. They want to know about fuel consumption, but for the investigator, I can now track on a map, you know, where they are, and I could correlate the GPS data with toll data and putting this all together from various sources is actually pretty cool. Patrick.
Patrick Bland: I could try uncheck the mute button over here. Yeah. So in coming in and just for throwing this out there and I know Warren said,I’m specifically speaking about Splunk technologies, I’ve been an engineer there prior to joining Acres. Splunk is not the only data platform out there. It’s a good one. There’s other good tools out there that kind of follow the same type of construct, right? And these might be tools that in the cases that you’re working on. Your customers, your clients might have them within their environment. So it’s something to keep in the back of your mind.
But in terms of getting this data in, I mean, there’s a number of different mechanisms to get the data. You know, the sensor data coming from the manufacturing floor could be going through like a kepware, or one of the ICS systems that Norman was talking about. There’s push methodology, so data streaming directly from the device to some kind of analytics platform. There could be pool based type data pools where that the platform could go in and programmatically script out a pool from a database on a specified period or reading data from log files on a specified period.
The whole point I have in the slide is just to illustrate how many different ways there might be to get a data source into another platform. Because I know traditionally in the e-discovery context and forensics context, we’re typically dealing with static data sets. And so that’s where, that’s one of the interesting things about these other data platforms is we’re dealing with streaming data coming in real time to do investigations, do alerts. And then, you know, going back to the more industrial kind of use that Norman’s been talking about, that’s one type of use case, right? For these types of platforms.
And I’m going to share my screen. Let’s see here. Right, I know we’ve been talking about explosions and hackers, infiltrating water systems, but you know, I figured we could talk about making data delicious for just a few minutes and lighten the mood just a little bit. Unfortunately, this is what happens when you get a bunch of data nerds together that love to cook and do barbecue. And talking about being able to leverage IoT sensor data for any type of use case, right? So I had a colleague of mine that lives in Kansas City that has basically built a smoke monitoring device with the little Raspberry Pi that he’s got in a number of the large Kansas City smokehouses, right? And there’s actually, he’s discovered a business benefit to this.
So instead of, you know, grandpappy saying, hey, every four minutes we throw four logs, or every 30 minutes, we throw four logs into our smoker, right? By actually using the data coming off of the sensors, they’ve been able to tell, well, maybe we throw a log in every hour. So they’re seeing a 25 to 30% decrease in natural resource consumption just by measuring the quality of the smoke that’s coming off of these smoking rigs. And just to give you an example, and I’ll show you kind of a live action demo here, but this is just showing you the type, you know, one possible example of what IoT data could look like when you’re actually taking a look at it and doing an analysis.
Right and so here is a little home Splunk instance that I have set up. So I’m going to walk you guys through just some quick samples of some sensor data that I have in my house. So I’ve got a little Splunk server here that’s capturing home automation data from my Smartthings hub. And here I’m just going to run the search again. So we’re just taking a look at the last four hours and let me move this video so you guys are out of the way. Yeah.
And here we can just see some pretty nasty kind of JSON data, but there’s a lot of information that’s coming in from these events off of these various sensors. Right, so if we break this out, we can see, you know, this is a front door sensor that I have, it’s giving me the temperature right now. And depending on the device, there might be different polling periods. We can see if we wanted to break it down by, okay, I’ve got my little office sensor right here, show me all the event data from the last four hours that I’m getting from my office sensor. Right and so you know, we can see additional data about here.
So, I mean, this is just the raw nasty bits that are actually coming from the source. And so you know, one of the big questions is; hey, what’s the organization? How are they using this data? Right? How are they making decisions based on this data? Right and so I might not be interested in doing analytics or doing an investigation on the raw nasty bits here, right? There might be something that I’m wanting to actually leverage a chart. Right and here’s just a quick little time chart of the raw data we’re just looking at. And then I can layer this on and give me a nice little graph of the event count by the different devices that are reporting in my home.
And also another thing. Right, and like I was mentioning earlier, right? Just the ability to do this on a real-time basis as well. So I’ve got a sensor, hopefully the motion thing will pick up. But this has given us a 30 second window of the actual devices that are sending data through… grab this guy. Here we go. Sometimes it takes a little bit to start up. All right, so this is the little sensor that I have next door to my computer. Right here it’s showing that, hey, there’s an acceleration event, there’s a polling period on this every 15 or 20 seconds. I mean, so you can actually see data streaming in. Right, so here’s the humidity reading from this device here.
So, a lot of different ways to make use of this data. And then going back to the whole security angle of this, especially from a home consumer standpoint, you probably don’t want to be buying, you know, unknown door knobs or smart locks and stuff for your house from a vendor that doesn’t really have any security protocols or anything in place. Cause it could be an entryway for a hacker to get into your network. Right and just going back over to show off another kind of lens into this data. So this, taking a look at the IoT data that I’ve got in my house, just in kind of a static dashboard type mode. So here we can see, you know, just the event counts by the sensor. So we can see that my office sensor’s fired off the most events because I’ve got it tuned to send events more frequently than the other sensors in my home. Average house temperature. Apparently my office temperature is… The sensor’s off a little bit because if my house is 641 degrees I think we’d have bigger problems to deal with.
You know, so there, there’s a lot of different ways that you can visualize and do analytics on this. Right and then one of the last things that Warren was talking about is, you know, these data platforms that are out here are great investigative tools, right? So they do have, if, let’s say, if you do find a data set that you want to export out to another tool for additional processing or analysis, you know, a lot of these tools do have export capabilities where you can get the raw data out. Right, if you wanted to export but also a lot of the platforms that are out there have pretty good and pretty advanced query languages that you can get a lot of value out of the data that you’re looking at, right? Or really do a deep level investigation, the data you’re looking at right within the tool itself.
And then last but not least, going back to the industrial control side of the house. Right, this is just another example of being able to leverage IoT type data within your clients’ or your customers’ environments, that they might have systems like Norman’s, they’re probably having systems like Norman’s that are there. And then just like what I’m showing you guys, here’s an example of Lockheed Martin where they’re actually using, Splunk is one of their platforms or tools to get data from their ICS systems on their floor for like predictive maintenance type use cases and being able to keep the health of their systems online so they can keep working on the the new F- 35 fighter jets. So a lot of very interesting things that you can do with data once you’re capturing it. And this could be on-prem or it could be in the cloud, like going back to what we were talking about before, where’s this data going to live? And so that’s something to keep in mind when you’re trying to track it down.
Warren Kruse: And to Patrick’s point that he just made was pretty good about, you know, what you could do with the data, if it’s being stored, but as, you know, a consultant that, I’m sure others that are viewing are consultants as well,or maybe you work internally and you don’t know that Splunk’s being used for this. Knowing about the existence is what, you know, hopefully we’re sharing with, you know, to broaden our horizons beyond, as I said earlier, you know, it’s not just the thermostat on the wall. It’s not just the doorbell or the camera that’s the IoT. I mean, companies are running on this data, running, using these devices and the data is being stored someplace. We need to find it.
Patrick, other than just saying, hey, do you use Splunk? If someone walks in and they’re trying to help an organization, what are some keywords or who should they be talking to to find out if they’re using these types of tools?
And Norman, you know, you as well, any keywords or anything that they should be looking at, you know, we used to do custodian questionnaires and we would say, do you use email? Well, yeah. Everyone uses email these days. Do you use a cell phone? Well, yeah. You know, what kind of, what can we be asking for is my question.
Norman, you’re on mute.
Norman Rankis: Sorry about that. And within the manufacturing world, I think the key thing is first understanding what your architecture is, what your network is, looking at the switches, talking with the corporate IT team as to what type of perimeters are they using like logic monitor and you know, various other types of tools to help harden the attack surface. And then start taking an onion layer approach as to if you’re working with the IoTs, what are they connected to? How do you update those? What’s the firmware update release? What’s the most current release? In other words, onion layer approach on the key devices. Also working with just not the IT people but working with the electrical engineers that may be working with PLCs. How do you update this? You know, how do you download? How do you upload the product?
Also looking at the manufacturer if you’re looking at, say a corrupt software, that’s been changed a lot of times, there’s footprints. For example, Rockwell will have the log-in name of the operator that uploaded or downloaded that software, that will give you, you know, a tip.
So basically you know, onion layering it, turning every single stone as you do in a typical forensic investigation group and putting together. If you have the data, as I pointed out you know, trying to map out a historical trend as to what happens of the settings of a particular machine or, instance, let’s say if you have a machine that makes Coca-Cola cans you know, what was the speed setting at of the conveyor belt? Oh, look, it changed from, you know, from 60 to 200 you know, who was the operator that oversaw that and started getting the operators started asking those typical questions of occurrence much like what –
Warren Kruse: Ask questions and you know-
Norman Rankis: Yeah.
Warren Kruse: Ask questions and listen to things that hopefully investigators can do a lot of. I want to ask Trey, I was talking to Trey about my problem, my personal problem with my alarm system and my camera. And I said, Trey, how do we get all of this data from people’s personal IoT in and make use of it? So I don’t know, Trey, do you want to talk a little bit about some of the stuff you’ve been doing?
Trey Amick: Yeah, absolutely.
So, I know we’ve been talking about industrial you know, pieces, but trying to relate it back to maybe something, you know, people might see a little bit more often, especially when they’re looking at, you know, mobile devices and what’s being seen. I know we talked about it a little bit earlier, as far as, you know, what’s being stored in the cloud versus what you know is going to be local on that handset. You know, let’s talk about Ring for a little bit.
So obviously, Ring was acquired by Amazon back in April of 2018. So when we speak of Ring, we’re speaking of Amazon, so right off the bat, we know there’s probably gonna be a lot of cloud data, you know, for that. What I wanted to highlight here, if there is some good news, especially for law enforcement, there are, you know, is access to logs and the community opt-in programs that Ring provides, where once you have that access, you can glean you know, potentially other videos from other sources that are kind of all being compiled in that community program. Kind of show you what that looks like here in just a little bit.
And then down here in the screenshot I took, the BBC actually made a data subject access request or DSAR into what intermission Ring weight collects. And the period of time that they collected from was the end of September of 2019 through February of 2020 for a doorbell. And then at the end, they added in the video system as well. But for that, you know, fairly short time period, there were over 1900 individual events being recorded and captured in this log. You know, things from motion being activated by the sensors as you can see the time column, to things such as the ding, which obviously is the doorbell, piece of that as well. So you’re going to have an idea of what’s going on along with, you might see one that says on demand, which is going to be somebody, you know, remotely accessing the Ring doorbell to view what’s going on or to speak to somebody outside. And then obviously that event ID that’s pretty helpful as well.
Now, obviously that’s kind of what you’re seeing in some of those logs, I’m going to share my screen very quickly here. Give me just one second. And there we go. One, can you see my screen? Okay.
Warren Kruse: Yep.
Trey Amick: Perfect. So, what I want to show, I’ve got a iOS device that I made a file system level image of looking at that Ring doorbell, just to see what we can get. And one thing I always want to point out to people, you know, look and see what people are doing on their devices. That’s going to tell you on the apps that you might need to dig into to start. So I’m actually going to go into operating system here, just scroll on down to the knowledge to database, which obviously, you know, you’re going to get 28 to 30 days of a snapshot of that device history and give that pattern of life usage, which is so critical for, you know, really pinpointing the different apps that you need to dig into here.
So that being said, I’m going to look at the knowledge application you know, usage piece. So give me one second. Application usage. You should, and as I’m looking here, this is a great start to understand different applications that you know, that user is touching. So we can see, you know, past books. So they might’ve been in their Wallet, Weather, Instagram, Mail, Facebook, Starbucks. Okay, keep going down. And now you start seeing things like Ring. So that’s telling me right off the bat, okay, this is something I might want to dig into a little bit more. So, you know, a good first step of when you’re looking to figure out where you need to pinpoint that investigation, you know, for digging into that.
Now, other places that we might want to look into, especially when it comes to Ring and what’s being stored on this device is actually going to be within the knowledge as well. And that’s going to be the knowledge, say notification usage, which is down here. Now, why is this important? Well, Ring gives you those notifications, if somebody is at your door or rings your doorbell, things along those lines. And as we’re looking through here, you know, what can we see? Well, we have Apple shortcuts. We can see low power mode. We do say we have notifications from Ring as well. Now, the reason this is important is because what’s being stored on this device is often just going to be from the cache of what is going on and what you’ve interacted with within that app. So you’re going to have an idea based on the notifications that, hey, maybe they click on it, they were able to log in and that cached something to this device that could be important.
The other thing I would want to call attention to very quickly here is the apple.shortcuts. Now, this is obviously an app that’s built within iOS now. Its predecessor was the third-party app but if this and that, that’s something that a lot of IoT devices kind of connected to in the early days. And essentially you can build, you know, if for instance, my thermostat goes above 70, you know the AC will cut on or, you know, you can build more complex ones. If my AC comes on, then I want my lights to cut off and you can keep building out these workflows. And shortcuts does the same thing. So that tells me right away as well from here that, hey, they’re probably doing something within shortcuts I might want to dig into.
Now to find that Ring data, there’s a couple different ways we can do this. I’m just going to go up to mobile very quickly here and we can go into the installed applications. Now, a couple of quick ways of being able to do this here. We could just, and I like to move this display name over to that package name so I can see what’s going on from this point. We could just start scrolling through, looking through. We could actually come in and build a, you know, a complex filter saying, hey, I want to remove anything from the com.apple because Apple uses the reverse domain structure. So you see only the third-party apps or even easier, you can just do a filter on the column of the display name, just type in Ring. So if you want a full listing of all the third party apps, you can do that. Or if you’re looking for something very specific, you can use this as your roadmap. And here we can see we have Ring, we definitely have it installed, when I select that information. This is your roadmap now to find out where that data resides on this device. So we have to do it that we can copy and paste.
Now from here, I can just hop over to the file system view to start looking at that. And when I do that, we can start seeing what is embedded within the data structure of this app. And when I’m scrolling through here, I’m going to kind of start at the top here, as you can see, I’ve already got the filter turned on for this. And, you know, as we’re going through here, the first thing that kind of caught my eye when I was looking at this is this phone 1 log. And I got really excited about it thinking, oh, hey, this is going to be a log of all the interactions, everything that’s going on. Unfortunately, it’s not going to be that exciting. It’s going to have a lot of just noise as far as what’s going on on that device at any given time. It’s going to have like a, you know, it’s going to be checking the network and making sure the speeds of the network you know, from time to time and just basic information in there. So not super, super helpful when you’re really trying to get an understanding of what’s going on.
But when you kind of keep digging through here, we can see the com.ring plist, which I always tell people, if you’re looking at Mac or iOS, always look for that main plist that you can start digging into, you know, to get some information potentially about the user count or what’s going on. And it’s going to have things, you know, I’m going to pop this out so we can see a little bit better here.
You know, the last time that, you know, there was an interaction with you know, the Ring and, you know, some of those bits and pieces. So I always tell people, kind of check this to see what’s going on. You can also see here, you know, with the shortcuts and the new shortcuts, all of these are showing as empty, so there’s nothing else really being used from within Ring. But just good information to have.
You can also see here the ding tracking service. Now, if you notice that number is eerily similar to the number that was in the event ID, from what I was showing on the PowerPoint earlier. So you kind of have a event ID that you can start tracking back from with some of this information. And as we keep scrolling down, you can also see, hey, we have the ID and we also have a date of when that happened, so once again, starting to narrow your investigation down.
Now for law enforcement if you are requesting information from Ring specifically, really the fastest way to get reliable information is have the Mac address if at all possible of that device. And you either go into the Ring application of your, you know, victim or suspect and go under devices and pull that information. If you have access to the Ring device itself, you can look on the back of the doorbell, the security cameras. The Mac address is actually located near the power cord near the rear mount as well. Or if you can find the box as well, that’s always great, but that will expedite you getting that information back and making sure it’s the right information as well.
Now, a couple of other pieces of information I want to scroll through here. You know, as we’re scrolling through, we can see we have an MP4 here. Once again, we can come in, we can click play and something that’s important here is the community aspect that I was talking about at the very beginning of, you know, allowing to opt in to this. Well, this video is actually part of that community alert system. And I did, in fact, look at that video on the device, under the community setting, and it did cache to my device. But this is the type of information that you might be able to pull as a law enforcement officer, you know, from the community-based aspect of this. But once again, it did cache since I looked at it on my device.
Now, as I keep scrolling through here, we do have the cache key, once again, thought I might get lucky when I was looking at this thinking, hey, I got the full, maybe, database of information that could be important in here. And you know, when I opened this up, you can see there’s a lot of binary plist. Now we could always right click and do those plists as well. But honestly, not too, too much there.
Looking at some of this other information, we can see, you know, the cache response list, you know, and see what some of that information is. And this could be handy because this is actually showing some of you know, the searches and what I was looking for and kind of digging into. Now obviously I have tried pulling out some of these URLs to see what would happen. Didn’t really go anywhere, but you do get a timestamp and getting some idea of some of the integrations and some of the alerts that are happening off this device. Here’s a video search history and when I did that as well. So a little, you know, a little bit of information to kind of start piecing some of this together.
As I keep scrolling through here, you can see got a png you know, that selected. Now, obviously some of these you can do quick previews on. Here you can see it’s just a quick snapshot of what was being viewed at that given time on that Ring doorbell, when you activated it live. You can see there’s a lot of pictures and videos on here as well. And, you know, keep going down through. Here you can see there was actually an alert on the device. I clicked on the notification, what happened? It went to a live view and I got a picture of the kid making faces at the doorbell, which I thought that was pretty funny, cause that’s what it cached from the live video.
You know, as we keep looking through, as you can see, there’s a lot of data that’s within this app set, and then we have a bunch of these files. And when I click some of these, you can see, this is actually going to be a JSON and we do have a built-in JSON viewer from within AXIOM. And when I open this information up, what can we see? Well, we have timestamps, we have the ding ID. So once again that alert that we can cross-reference for, if we are having to collect information from you know, directly from Amazon/Ring for this. So you can see what’s going on, you can also say, hey, did they favourite that video? Because when you log into the Ring portal online, you can favourite videos. You can see the duration of what was going on there as well. And then you can see, hey, was this alerted through Alexa? Was this an auto-reply? Did they find a person as a part of this as well? So, you know, some pretty good information that you can cross-reference with the ability to pull some of those additional pieces when you get your search warrant or when you do a collection and you log into that Ring account because you can see a lot of this information when you log in as well. It’s just not saved directly to that device.
So I always recommend kind of looking through and, you know, as you can see, we have a lot of JSON files here, but making sure that you’re kind of digging into that so that when you go to request information from these providers, you can be very detailed as far as, hey, you want that specific ding ID and as much information about that as possible with this. But that’s definitely just a you know, piece of what I wanted to show today, but Warren, I’ll turn it back over to you.
Warren Kruse: Awesome Trey. Thanks, Trey. Appreciate it. I hope that the attendees, you know, like we said earlier, we were here to raise the awareness beyond IoT just being for home use. We will now jump over to the discord for some Q&A. Please join us there. Thank you for joining the presentation. And we look forward to seeing you at the Discord. Thanks everyone.
