Even in an ever growing cloud-based world, some companies continue to anchor their resources firmly to physical on-premises (on-prem) locations, defying these cloud-based trends. While using cloud based infrastructure to host applications brings ease of deployment, scalability, reliability, and the flexibility of offloading the application to the vendor; companies still opt for on-prem deployment options for the control, data sovereignty, predictable cost, fine-tuned performance, and security oversight abilities that come with hosting your own applications. Companies that choose an on prem solution generally do so because they have the resources, both financial and human, to deploy, manage, and administer the application.
A company I worked with previously had been practically grandfathered into on-prem deployments through decades of business, and even with the prevalence of cloud technology in the modern world, they continued to move this route for future deployments as well. Leadership liked the control over the configuration and management of their products, possessed the physical space to store the hardware to host the applications, and enjoyed the predictable costs outside of the upfront costs. Most of all, the organization wanted to have direct control to tailor security policies on applications to their needs. On top of that, being a company operating world-wide in 50+ countries with approximately 40,000 employees, they had the personnel available to hire new Subject Matter Experts (SMEs) to manage and administer each newly deployed application, or to appoint an existing employee as admin. But, as the organization started to become more security focused, the team quickly realized there was a major problem: Due to the growing size and complexity of the organization, visibility over the environment was becoming increasingly complicated and difficult to manage.
The Problem
As the security team was rapidly maturing, the team first noticed there was no single source of reference that could be trusted. The Configuration Management Database (CMDB), which was supposed to be the trusted database, had not been accurately maintained for quite some time. It was also known that there were systems floating around unaccounted for in the corporate environment, and M&As (Mergers and Acquisitions) had many devices that had not been onboarded due to no SOP (Standard Operating Procedure) for documenting new assets in the network. The company has been using so many different services to manage all of the devices in the environment (load balancers, firewalls, servers, workstations, etc.), and due to Acquisitions over the years, there was never an effort to assimilate to a singular system. This was quickly becoming an increasingly difficult problem to solve by the day. The company had become so large and complex that no one, let alone the security team, had any idea what assets were in the environment. What systems are in contact with each other? What devices are outward facing? In the event of a Security Incident, the Incident Response team would have no idea the scope and impact of an attack.
The Solution
A few years back, I was tasked with tackling this problem. The high-level solution was defined as a dashboard a user could interact with the network traffic data through. This dashboard was to be used by the Security Team to triage incidents and understand the scope of a breach, or by IT to eliminate unknowns and ensure that ingress and egress traffic was moving as expected. Let’s break down the pieces and how they all fit together.
The first order of business was to identify every platform that was used in both the corporate and M&A environments to manage all the different network systems. This process involved first reaching out to 20+ M&As of various sizes to determine what they used to manage their network devices, and then to figure out how to pull these configurations via API. Pulling the configurations via API ended up being the easy part. The difficult part proved to be, first, finding someone who knew what was going on in the environment to point me in the right direction, and second, to get the necessary access needed to pull configurations.
After knocking out the M&As, I turned to the corporate environment. This turned out to be much simpler to get access to all of the necessary admin panels, but writing the code to pull the configurations via API was becoming tiresome at this point. Due to the size of the company and poor use of documentation while scaling, it was often unclear what instances were used for, or even if I was dealing with legacy systems in place no longer being used. But finally, after just a couple of weeks, I had finally written the script that was able to pull down all configurations (both corporate and M&A environments) to create a basic inventory of all the network hardware.
Last on the list of assets to track down were the workstations and servers, which were fortunately pulled via an Asset Inventory solution. While everything was thankfully gathered in one platform here, trouble this time ended up coming from the complexity of the API and poor documentation (I won’t name the system here, but if you’ve used this Asset Inventory tool’s API before - you’ll know). After some trial and error, I finally had everything I needed in order to start mapping out the organization.
The first process to mapping the environment was to write the script that would handle mapping out the traffic from externally facing assets to internally facing assets. With the knowledge I had, I made the assumption that there are 3 types of mappings that could occur in the environment:
External traffic hits a firewall, then a load balancer, then an endpoint.
External traffic hits a firewall, then an endpoint.
External traffic hits an endpoint. (Hopefully nonexistent)
Once I got traffic mapped to an endpoint, I would then loop through the list of applications to figure out what (if anything) was hosted on that machine.
Let's do a quick recap of where we are so far: My script starts by pulling down configurations from each data source via API and classifying each asset into an object of firewalls, load balancers, and endpoints. Then the script iterates through all of the load balancers, and adds a row to a temporary table for each connection from load balancer to endpoint. After drawing these relationships, the script would now iterate through the list of firewalls, drawing connections to any previously established load balancer/endpoint connections that fall within its subnet range. After this list had been exhausted for each firewall, the next step would iterate through all of the endpoints to find if there were any direct firewall/endpoint connections. After this, options 1 and 2 from the table of possible object mappings above were complete. The final step was to look for any mapping types for number 3, essentially searching for endpoints with public IPs. Luckily, there weren’t any.
This first portion of the script handled the mappings of any ingress traffic, but there was still egress traffic to map out. I applied the same mapping methodology here, but in reverse - mapping servers back to their load balancers if applicable and firewall counterparts if outbound rules existed. There proved to be a much simpler architecture here for the few outward communicating services.
The last step of my script was to iterate through my final table of mappings and append the name of connected applications to any machines hosting one. Luckily, the difficult and time consuming portion of identifying and mapping all applications within the corporate and M&A networks had already been completed by the team. So, all that was left to do was pull down the application information via API and iterate through the table of mappings, appending the extra columns whenever relationships were drawn.
To finish off the solution, this script was run on top of a cron job that would update the table every 6 hours. The table was connected to a dashboard in a simple table format to make it digestible for the team. It allowed users to view the traffic as if it was flowing left-to-right and sported a plethora of filters to narrow down search results, including:
Ingress, Egress, or both.
Search by device name for any device category (regex handling available)
IP subnet range support both by device category, and as a general search
Checkbox Dropdown to search through all applications
And more…
The dashboard also had some basic analytics populated below the table view to give total connection counts, counts per object mapping, and change metrics over time. This solution was able to provide a holistic view over the ever growing network.
The Impact
The dashboard ended up being primarily used by the Incident Response team in order to approach security incidents with much improved network visibility and to better understand the impact. The team could now easily identify network assets, draw connections between systems, and mitigate threats quicker by being able to identify the blast radius instantly. It also helped the Security Operations and Engineering teams work to enhance the security posture by looking for potentially misconfigurations, unauthorized access points, and identifying strategies for network segmentation. There was also opportunity here for the tool to be used by the IT team to optimize load balancing, resource allocation, and overall spend, but this was not something I was able to see through after finishing the solution for the organization.
Overall, in an ever-growing cloud reliant world, this was an interesting, hacked together solution to solve a problem of network complexity for this primarily on-premise organization. My contributions helped the organization make better informed decisions, reduce attack surface, monitor data flow, improve their incident response processes, and gain a holistic understanding of the data flow for the applications in their network. It was a significant step towards ensuring resilience and integrity of their network infrastructure.
Thanks for joining us in the Cybersec Cafe! We hope you’ve enjoyed this conversation on a custom solution to internal network mapping. Stay tuned for more insights, updates, and experiences in the ever-evolving cybersecurity landscape. As you step out, remember to keep sipping on knowledge, sharing in wisdom, and above all, securing your digital world.
Comments