Editor’s Note: I wrote this in July of 2021 but forgot to publish it. It’s July of 2022, so I’m going to make it public without any edits. I’ll write another post about my longer term experiences with this transition.
It’s been a little over two months since I started my new job. What a wild two months it’s been. But since this is the first post, I should provide a little additional context. Which is weird for me, because I’ve largely avoided talking about where I work or where I’ve worked to the general public. But here we go.
Backstory
I began my career working on solid state drives as an intern at Intel in 2013. I was responsible for power, performance, and thermal analysis for consumer SSDs. I was there for about 15 months and I had a lot of autonomy to do my job. I didn’t know much of anything about SSDs, other than that they were faster than hard drives and that I wanted one but couldn’t afford one as a broke college kid. My manager was a principal engineer who had worked in the flash memory industry since 1987, and he hired me after a 45 minute phone call where we talked about why I liked building my computer, what upgrades I wanted to do next, etc. He gave me a lot of room to grow and he taught me a ton about a domain that I knew very little about.
After my internship, I finished school while doing a brief stint as “CEO” of a small 3-person security firm. Mostly I handled business stuff, since I had spent a couple years taking business classes already. Then I handled a bunch of freelance web dev stuff instead of doing security stuff. When I graduated, I went back to Intel, this time doing Quality and Reliability engineering for Intel Optane drives. After about a year of that work, I started looking for work in security and that’s how I ended up meeting the head of Intel’s red team at the time.
I volunteered on Intel’s red team, technically “20%” of the time, but ended up being more like “20% of the time and also 50% of my after-hours time” because I was really enjoying what I was doing and I really wanted to make the full time transition. I wrote a lot of tools that did what we needed. I wrote a lot of documentation for existing tools. I was a big fan of writing documentation, and I joke that that’s how I got into my red team job. Eventually I did pivot into red team full time at Intel, where I stayed for just about 2 years. Red teaming at Intel focused a lot on the company as a whole, much like you might expect when someone says they work on a red team.
After Intel, I went to Oracle Cloud Infrastructure’s (OCI) red team. I spent another 2 years there, this time focused entirely on the security of Oracle’s new cloud play. OCI was full of very talented people and built a very strong foundation, which meant that a lot of more traditional red team approaches were very difficult. I learned a lot about cloud architecture, huge systems, and the challenges of building out a red team function at an organization that maybe wasn’t entirely ready for that function yet. I also got to work a lot on our scan platform, as well as work with our Detections and Response team on improving detections and response capabilities.
I left OCI’s red team in April 2021 to join a startup as the first security engineer.
Culture Shock
While I do want to talk about my experiences going from Red Team to Security Engineering, I feel like it’s important to first address the much more obvious change – I went from Intel and Oracle, large multinational tech companies with hundreds of thousands of endpoints and a hundred thousand employees… or more, to a small organization with about 150 employees and two offices. Talk about a culture shock. Things that took weeks or months to get approved or rejected at my old jobs would have a decision made within a day or two at my new job. I saw problems at my old jobs that I wasn’t necessarily able to fix, either for political reasons or simply because I had no access to make the necessary changes. Meanwhile, I have been able to jump right into this smaller organization and make changes across application security, infrastructure security, corporate security, and even employee privacy.
I spent years at large companies because, frankly, I was afraid of instability. I was afraid that I’d lose my job for one reason or another and wouldn’t be able to make ends meet. I grew up quite poor and now that I finally had a steady income, I was afraid to lose it. I didn’t have savings yet. I still had a lot of debt. I was living far away from home and didn’t have a safety net to fall back on if something went wrong. But somehow, during all that time, I also watched numerous large layoffs that came without warning and affected large portions of the company. I watched at both companies as my performance was quietly recognized, but not rewarded. I chose these companies for the perception that I would be more stable in them, all the while being shown repeatedly that the stability I was looking for wasn’t necessarily there.
Finally, I had enough saved up that I could afford 6-12 months of unemployment if it came to that. I wasn’t specifically looking for a small company to go to, but I knew that I was looking for something else, culturally speaking. I wanted a role where I could take more ownership of things. A role where I could have broader impact. A role where I not only got to find gaps in security, but got to drive them to closure. Red teaming was a lot of fun and I worked with a lot of smart people and learned a ton from each of them and from each engagement we did. But I felt like I wasn’t going to get what I was looking for if I just kept doing engagement after engagement, finding a long list of things that could be better, and then moving on to the next thing. I needed something else.
Security Engineering
I’ve always been a big fan of being able to build solutions. I used to look at products that we’d bring in at Intel or Oracle and think “Why are we paying so much money for this when it has these obvious shortcomings?” Then I’d think about how to achieve a similar objective with something in house, and why that expansibility would ultimately be better for the organization. But that line of thinking definitely didn’t work when I got to my new role. No longer did we have a hundred or more security engineers, processing the results from nessus, trying to hunt down systems to get them patched for each new hot vulnerability, building dashboards for ingesting all the latest data, etc. We have me, an IT person, a compliance person, and the guy that manages us all.
I still look at problems and I think about them in terms of how I would solve them if I were building a solution. But once that’s done, I have to think about how long it would take to build the solution versus how long it would take to integrate some other existing solution, versus whether or not it’s even a priority compared to what else I could be spending my time/money/energy on. In red team land, I didn’t really get a firm understanding of how priorities competed with one another, other than that I knew they did and that’s why things wouldn’t get done. But now I’m seeing it first hand and it’s been extremely enlightening.
Being the first security engineer at the organization, I found that my work splits up into a number of fairly distinctive domains:
- Application – Typical appsec type stuff. An XSS in the application. A CSRF over here and a SSRF over there, here a SRF there a SRF, everywhere a SRFSRF.
- Infrastructure – Focused on the security of the infrastructure, such as stricter access controls, auditing, patching, finely tuning IAM roles, etc.
- Corporate – Endpoint security & visibility, security of employees, SSO policies, MDM security, incident response, etc
- Culture – Publishing security advice, encouraging good employee security habits, security office hours, a public vulnerability disclosure program, etc.
- Product – Introducing security features and improvements into the product we provide, such as additional MFA options, SSO support, etc.
Some weeks have leaned much heavier than others on one of these domains, while other weeks are a hodge podge of all of them. It is sometimes difficult to juggle them, however I’ve found it infinitely more rewarding to do so.
Just do it
Perhaps my absolute favorite thing about my new role is that I don’t have to spend a lot of time trying to convince people that what I want to work on needs to be worked on. I don’t always need to convince people to allocate resources to make a change I want to see. I see a thing that should be improved or could be improved, and I have the ability to just go do it. I made my first change that got deployed to production within my first week on the job. I’ve made a plan for several weeks worth of work revamping the product’s authentication story to better align with ASVS, which I’ll be working on in the near future. I’ve taken charge of rebuilding our application containers to be more efficient while simultaneously reducing our attack surface.
Contrast this with my work on OCI’s red team, for instance, where I largely wasn’t able to just do things without first getting explicit approval from one or more managers, and would be seen as stepping on toes if I volunteered to do something that was our priority but not their priority. This new role encourages outcomes without getting so in the weeds about who makes it happen. It’s the parts of “extreme ownership” that I actually agree with, and that makes it incredible.
The balancing act
One thing that I struggle with, and that I see my peers struggle with, is the balancing act between what is the long term good solution and what is the short term “good enough” solution. A lot of security gaps, in my opinion, come from these decision points during the life of a company or a product. When something needs to get done sooner rather than later, corners get cut to get to “good enough.” There is hopeful talk of a time called “later,” where the company has more time to fix the things that they know they need to fix, but that time never seems to come. I’m extrapolating here, as the same “later” was often talked of in the much larger security organizations I was previously a part of, but still it never came. 50 billion in annual revenue with some 300 security staff? Still no time to fix the fundamentals. Later never comes.
So it all comes down to balancing the short term and the long term. The finite and the infinite. How do we make sure that the short term solutions we’re implementing today best align with the long term solutions we need to ultimately be successful? How do we make sure that our long term solutions don’t stand in the way of making progress today? This is a question that exists across the company as a whole, not just squarely in the security domain.
For security to be successful, I think we need to be able to inject ourselves into this balancing act and be responsible for driving the solutions within that balance. My experience with infosec at Intel and Oracle have mostly left me with one key takeaway – security needs to be much more integrated into the company as a whole, rather than tightly concentrated into organizations whose whole role is to bark about policies and demand work from others. Get out into the organization, offer to fix things, teach others how to think about security the way that you do, learn from them about how to think about business requirements the way they have to.
You might say “But dade, there are so many things! It’s sisyphean!” To that, I tip my hat in your general direction before going back to building the tools that make it easier to push the boulder, and the platforms necessary to prevent it from rolling so far down the hill next time.