[EXHIBITOR EDITION] Cybersecurity Isn't Just an IT Problem

Guest: Andy Lomasky, Senior Director, IT, PMMI

In this episode, cybersecurity expert and PMMI’s Senior Director of IT, Andy Lomasky, joins unPACKed to explore key challenges and strategies in protecting industrial systems. We discuss network segmentation, remote-access risks, and essential steps that OEMs can take to embed cybersecurity into machine design from the outset.

Sean Riley: You're listening to unPACKed with PMMI. Real data, real innovations, real strategies to move your packaging and processing business forward.

Hi, and welcome to unPACKed with PMMI. I'm your host, Sean Riley. In this episode, we welcome cybersecurity expert and PMMI Senior Director of IT Andy Lomasky. He joins unPACKed to explore key challenges and strategies in protecting industrial systems.

We discuss network segmentation, remote access risks, and essential steps OEMs can take now to embed cybersecurity into machine design right from the outset. Let's have a listen.

So, with all the fancy introductions out of the way, welcome to the podcast, Andy.

Andy Lomasky: Thanks, Sean. Great to be here.

Sean Riley: Oh, the pleasure is all ours. It's great to have you back. Since I only have you for a limited amount of time, I want to try to touch on as much as possible. So why don't we start out with IT-to-OT attacks? How can an attack that begins in a company's IT environment, let's say a phishing email, suddenly reach the machines and production systems out on the plant floor?

Andy Lomasky: You're absolutely right, Sean. That's exactly the most common path we see for attacks jumping from IT to OT. It starts with a phishing email, or maybe a stolen password or a piece of malware that gets installed on a corporate laptop sitting on the regular office network, and the attacker is able to use any one of those things as an entry point to move laterally across a network.

And it's the fact that the network is not properly segmented that allows them to do exactly that: to jump from the IT side of the network, where corporate personnel might be working, over to the OT side of the network, where there are PLCs, HMIs, sensors, controls, and whatnot.

That's exactly how these things happen. They use an entry point on any part of the network they can get into, and they try to see where they can jump across to get to something else that's more impactful.

So they're generally not targeting OT directly on day one. They end up landing somewhere in the IT world and figuring out how to jump over to the OT side at a later point.

Sean Riley: Okay. So as machines become more connected, whether it's with MES, ERP, or the other cloud and analytics platforms out there, what can OEMs and manufacturers do to prevent that connectivity from creating an easy gateway, a pathway into OT from IT?

Andy Lomasky: The first and most obvious thing is proper network segmentation, right? That means having segmentation set up not just between IT and OT networks, but even within the OT network.

So each machine, with its set of different components like a human-machine interface, PLC, or sensors, should be in its own segment, and each machine should sit separately from the next, and so on. That way, for purposes of remote access, you can only get to the one thing the OEM would need access to, and as an attacker, you can't jump across those segments and access something else that you shouldn't be able to.

So that's one thing I would do. The other thing you mentioned is MES, ERPs, and whatnot. There are a lot of data pathways and connection points you can create between machines and corporate systems, or between machines and the cloud.

Each of those connections should really be purposefully built with that specific data path in mind, right? So if you're trying to upload production counts to an MES system, that integration should be designed for that specific purpose alone, with the permissions on that integration scoped just to those data points.

It shouldn't be an open-ended connection that gives full access to a whole suite of different data points, and it shouldn't necessarily be open both ways so that data flows back and forth freely. It should be properly scoped and locked down to only the things you're trying to achieve with that integration.

Sean Riley: Okay. That makes sense. We'll get into remote access in a second, because that made me think of something. But before that, I was thinking about ransomware when we were talking about phishing emails, and ransomware is usually thought of as an IT thing.

What does an attack look like in the manufacturing environment, and how can the consequences extend beyond computers to the actual production equipment, operations, safety, things like that? Or is that even a thing?

Andy Lomasky: I think it's actually a bit of a myth that it's only an IT thing. It very much affects the OT side of the house and can extend all the way out to the production floor.

So let me tee this up a little bit. If you have a computer on the corporate side of the network that somehow gets infected with ransomware, and that ransomware spreads across the network and makes it to the OT side of the house, what does that look like, right?

It means that HMIs, these human-machine interfaces, the PCs built into a machine that allow the operator to program it, do runs, set up parameters, et cetera, all of a sudden go dark. Or the HMI may start flashing an error message because the server it's connected to, which holds recipe data, parameter files, or configuration files, has become unavailable because those files are now encrypted. It could look like a SCADA system locking up or becoming unavailable, right?

So you'll gradually start to see components or whole machines going down, and services becoming unavailable. Screens may be totally blank, or they may just flash alarms or error messages.

But on the production floor, this is where it starts to look different. In the IT world, you may not be able to access your files, but you just shut your laptop off and maybe hand it back to IT. On the production floor, you've got a machine that's shut down.

If you're trying to get that machine back up and running, there's now a lack of synchronization between what the software thinks the machine is doing and what state it's in, versus the actual physical state of that machine.

So all of a sudden, if your software thinks you're starting at the beginning of a run, but the machine stopped in the middle of it, different servo motors might be in different positions, sensors may be out of place, or tolerances may be exceeded. Even safety systems, like interlocks, are affected, right? The software may think an interlock is in place or not in place, and you may not be able to safely restart the machine until you physically reset its state.

So I think a ransomware attack on the production floor looks significantly different from one on the IT side of the house that affects just a set of computers.

Sean Riley: Yeah, it sounds absolutely awful, to be honest with you. What practical steps can machinery manufacturers take to reduce the risk of ransomware? And if an attack does occur, how can they help customers recover more quickly?

Andy Lomasky: I think that's a fabulous question, Sean. There are a lot of things OEMs can do. One of those, as I mentioned, is segmentation, right?

As an OEM designing equipment, you should build segmentation into the way that machine connects to the customer's systems and networks right from the get-go. It should be designed with segmentation in mind so that it doesn't become a jumping-off point to other things it shouldn't be able to access.

The other thing that I think is often overlooked, especially as it relates to ransomware, is backups. And again, backups in the OT world look different from backups in the IT world. For a laptop, we just re-image it, and it's good to go again. But for a machine, it's a lot harder to have a golden image of what should be on that HMI or PLC to restore it to. And then, again, you have that disconnect between the physical state of the machine and what the software thinks it's at.

So it's important not only to have a good backup process and the right software images loaded and ready to go at a moment's notice if a customer needs them, but also to have proper documentation. You've got to document everything.

A lot of this gets missed, I think, during the installation of a machine at a customer site, but it's a really essential part of the process: documenting network diagrams, how things are wired together, and what systems or networks these machines are connected to, and having really solid asset lists and parts lists, right?

Just document everything you can. When you're recovering from a ransomware attack, sometimes this documentation isn't readily available until you can find your way to it. Having it on hand will help you recover so much more quickly than scrambling to locate these things in the heat of incident response.

Sean Riley: Yeah, 100%. Okay, that makes complete sense. We touched a little bit on remote access, and it's obviously essential for troubleshooting and servicing equipment. Especially coming out of COVID, it became a big deal, and people became a little more accepting of remote access. What cybersecurity risks can it introduce for OEMs and their customers beyond the ransomware we've talked about? Are there other things?

Andy Lomasky: There are, and I agree with you 100%. Remote access is essential, and I don't think CPGs should be saying no to it right from the get-go. There's a secure way to do it, and there are lots of solutions to help you get there, but there are some considerations and risks. For example, if you have remote access endpoints that are always on, that's a 24/7 door into the plant network, right?

Sean Riley: Wow. Not good.

Andy Lomasky: So the alternative, obviously, is to have remote access on demand, when it's needed. That's the proper way to implement it: you can flip a switch and allow the OEM to connect, so it's not always an entry point for someone who's trying to do something evil.

The other thing is using shared or generic credentials, right? Everybody's seen this at one point or another in their career: a single login or password that's used to access a machine. Sometimes OEMs use a generic set of credentials across multiple machines or multiple customers, or maybe they use a default password on a component that never gets changed.

That's exactly where the risk is, because if that credential gets leaked or compromised, not only is it very difficult to recover from, but it's impossible to audit, because you don't know who has it at that point, right? So using named credentials is a real strategy there. I'm getting into the second half of things you should do, as opposed to things you shouldn't do.

Sean Riley: No, that's fine. I was going to ask anyway. Let's get into it.

Andy Lomasky: Yep. And then, of course, there's always the risk that one vendor, or one machine, can be a pathway to many customers. It's not just one machine in one customer's plant giving an attacker access across that one customer's network. Like I mentioned, if you use one credential across multiple customer sites, you've now become a gateway into multiple customers' machines and all of their networks, right? That's what we call supply chain-style risk, where one entry point can get you access to a whole bunch of different things. It's a major payoff for attackers and very bad for CPGs.

Sean Riley: Yes. Yeah, that's amazing. I didn't even know that was possible, but you explained it very simply: just by using the same password, a hacker can get access to things they weren't even looking for.

Andy Lomasky: Exactly.

Sean Riley: Okay. We're in an industry where we want machines to last, and we have machines that have been in service for 10 or 20 years, or even longer. That life cycle has to make cybersecurity particularly challenging for PLCs, HMIs, industrial PCs, and other components. So what does that look like? Why does that life cycle make cybersecurity so difficult?

Andy Lomasky: Well, Sean, it's just like your computer at home, right? How often do you replace your own personal laptop? Probably somewhere around every three to five years, as technology takes its next major leap forward.

The problem with industrial equipment is that it's built to last for decades, not just three to five years. Meanwhile, vulnerabilities and security threats evolve over time, and as you and I both know, in the age of AI, those threats are now evolving even faster than they used to.

Even a three-to-five-year replacement cycle isn't fast enough to keep up with all of those evolving risks in the threat landscape, and all the different ways an attacker can get into the embedded PCs and equipment in some of this machinery.

So a lot of these components can lose vendor support or become outdated. I'll share a personal anecdote: I've been to PACK EXPO and asked OEMs what operating system they're running on their HMIs. Some of them, not naming names, are still running Windows CE or Windows 7.

Sean Riley: Oh no.

Andy Lomasky: And Windows 10 just went out of support from Microsoft, and everything has standardized on Windows 11. There are a lot of machinery interfaces using outdated operating systems that are still being sold out there.

That's exactly why this is such a problem. If you're using any sort of component, even one that has support today, and it's not designed to keep up with the evolving technology landscape, it quickly becomes a security risk.

I also think there's a tendency in our industry to say, "If it ain't broke, don't fix it," right? We don't want to replace something that's working, that's paid for, and that's been working for a long time. All it needs is good maintenance. From an operational perspective, I understand that. But it doesn't remove the need to keep up with the latest cybersecurity patches, fixes, and risks that come with running a machine for an extended lifetime.

Sean Riley: Okay. Would the documentation come into play there too? Because how would you even know what you're running and where?

Andy Lomasky: That's exactly right. I think documentation is a really big fix for that. And if you've been keeping up with the EU Cyber Resilience Act, that's exactly what the act is all about: creating a bill of materials for the components, parts, and software that go into a running piece of machinery, which you can use to audit and keep track of where vulnerabilities might come from.

Because, like I said, an HMI may have a whole bunch of different components and software, and it's just one piece of what could be a very broad line with several different machines and components built into it. So that asset list, that inventory, and all the documentation that goes with it will become essential to keeping up with these risks.

Sean Riley: Okay. You mentioned OEMs running old operating systems, so let's use that as an example. Say you're advising a machinery manufacturer that hasn't historically treated cybersecurity as part of machine design. They might be a little behind, but we want them to get started. What are the three most important steps you'd tell them to start taking now?

Andy Lomasky: I think the first thing, like we just talked about, is maintaining that bill of materials for all cyber-relevant components. I think that's really key.

Second is committing to a documented vulnerability disclosure and patching process. A lot of OEMs seem to think, "Oh, when Windows loses support, that's a Microsoft problem, not our problem." But if that HMI is an embedded piece of equipment in your machine, it actually is your problem. You're creating a supply chain risk for your customers if you're not addressing it.

So commit to tracking those components through their life cycle, including when they reach end of life or when patches need to be installed, and communicate that with the customer.

And you've got to remember, this is a partnership too, right? Cybersecurity isn't one-and-done at the time of installation. It's a continuing partnership that extends into the after-sales support life cycle as well.

And the last thing I was going to mention is to think about end-of-life planning for a machine, or even just for components of the machine. Bake that into the design up front, because it will make it a lot easier to replace those components without replacing the entire machine, which will help both you and your customers limit risk.

Sean Riley: Okay, beautiful. I love that.

So, to close, since we're both at PMMI: how is PMMI helping companies fight back against cybersecurity risks and attacks? What are we doing?

Andy Lomasky: We're doing a couple of different things. Number one, we've got our PMMI CyberHealth Blog, which I personally write. Every couple of weeks, we publish a post on a different cybersecurity topic specific to the packaging and processing machinery sector. So if you haven't been there, check it out at pmmi.org/cyberhealth.

We also have our MaX member share groups, which are a big component of members helping other members with their engineering and operations challenges. And you'd be surprised: cybersecurity comes up in almost every discussion we have in our MaX share groups.

We also have education sessions at our shows and at our member meetings. So there's a lot of content on this topic. We're always trying to come up with different ways to help our members not only be more secure themselves, but also help their customers in the end-user space be more secure as well.

Sean Riley: Beautiful. Well, that's a perfect button to put on our discussion, Andy. I want to thank you for taking time out of your day to bring us up to speed on all things cybersecurity. Thanks again, Andy.

Andy Lomasky: Thanks, Sean. It's a pleasure, as always.

Sean Riley: Thanks for listening to this episode of unPACKed with PMMI. If you liked what you heard, be sure to follow or subscribe on Apple Podcasts, Spotify, or wherever you listen. That way, you won't miss any of the industry insights coming your way.

While you're there, we'd really appreciate a rating or review. Want more? Visit pmmi.org/podcasts for all of our past episodes and additional resources. Thanks again for tuning in. I'll see you next time.

‍