On the Importance of Giving Back in the Infosec Community

Full disclosure, Chris Sanders is my boss at the time of this posting. He did not ask me to opine on the RTF or to write this at all. These are solely my opinions formed from my experiences. - Patrick

--

Pittsburg, New Hampshire is the northernmost town in the state and in terms of geographic area is the largest town in the country. It is about a 32 mile drive to traverse the town starting at the Canadian border and heading south until you reach the point where the Connecticut River begins becoming the natural state line separating New Hampshire from Vermont. You would think for a town of this size there might be a considerable number of residents. There are not. In fact it is not unreasonable to assume that the moose population outnumbered the people there by a factor of two or three to one when I was growing up. For the record, there were about 800 residents of the town when I lived there and attended the local K-12 public school which consisted of approximately 200 total students. I was one of 16 in my graduating class. Like everything, there are goods and bads of growing up in such a small town.

Many of the good points I was reminded of recently while attending the Security Onion Conference in Augusta, Georgia. First and foremost was the incredible sense of community. I can not think of another industry where people are so willing to contribute to something larger than themselves. While it was officially billed as a conference it felt more like a get together of friends. In fact the format and important discussions that happened reminded me very much of a town meeting in New Hampshire. For those unfamiliar, town meetings in NH generally occur on the second Tuesday of March. This is a day when residents of the town would come together, discuss issues surrounding the town and vote on lots of things ranging from school budgets to what kind of new plow truck should be purchased for the next winter. It was this same sort of spirit that was apparent in Augusta. People all heavily invested in Security Onion and our infosec community that had their own unique stories to share and thoughts for the future. This day of idea exchange closed when the mayor of the town, Doug Burks, gave his views on where he sees things headed and what to be looking out for. To have this sort of grass roots level activism so available to the people in our industry is something that sets it apart and makes it great.

Sometimes good and bad are intrinsically tied to one another. When I was a senior in my high school I had already decided that I wanted to go to college and major in biochemistry. I always loved science. Living in a very rural place unfortunately meant I did not have access to equipment and instruction more easily found at bigger schools. Because of this and its remote location, Pittsburg had a way of attracting teachers right out of college who were trying to get their first job even though the pay was terrible. It was thanks to this situation that we had a math teacher during my senior year who also knew how to write code. I happened to take a class he offered in Pascal programming and immediately enjoyed it. Being there were only two of us in the class I got a lot of great instruction. I never would have guessed that after two years of studying biochemistry I would decide that I really missed the logic and problem solving inherent in programming and elect to change direction and pursue computer science instead. Had I not been fortunate enough to have an opening in my schedule senior year of high school with a young math teacher who enjoyed programming I would likely have never gotten into infosec. This is kind of troubling to me because I wonder how many kids out there living in rural communities never even have a chance to get interested in this great field because the resources just are not there. They are lacking computers, equipment or simply people with the proper training to teach technical curriculum and inspire. This is where I think we need to leverage the same passion and ingenuity found at the Security Onion Conference. Perhaps we need to spend more time not just thinking about our community as those actively participating in it but also as a way to reach out to those students of today that we will need in our community tomorrow. 

Maybe adding insult to injury is the fact that today many jobs are increasingly concentrated in more urban centers. As high-speed Internet expands, decent paying security jobs which can be done remotely should be viewed as something of a way to let folks like myself stay in rural communities and earn a decent living. Unfortunately, the communities that stand to benefit the most from this rarely have the resources to reach the kids who could benefit. This can be overwhelming to think about at first but there are people and groups doing amazing things in this area that need and deserve our support. For instance, the work the OpenNSM group has done is truly exceptional in making infosec accessible to anyone with a desire to learn about it. Because of my personal story I wanted to focus a bit more on the work of another group, the Rural Technology Fund. The RTF was created and is led by Chris Sanders (yup, that guy). I won't go into great detail about the group as you can read about it yourself at http://ruraltechfund.org but it is certainly worth reciting their mission statement here:

The mission of the RTF is to help rural students recognize opportunities in technology careers and gain the education necessary to work in the computer industry.

This is a noble goal and helps address many of the concerns I have about kids in rural communities and lack of access to technology resources which they need now more than ever. I was really struck by the pervasiveness of this group just this week when I learned that a small town located not far from where I live currently in Vermont, is receiving a 3D printer courtesy of the RTF. Further reading at their website shows donations of robotics kits, Arduino kits, and raspberry pis to rural schools all over the country. Out of curiosity I asked Chris how the foundation was able to do all of this great work. Chris is a very humble man. He is not one to be in your face about the fact that the sales of AppliedNSM go in part to fund this endeavor. In fact, the Rural Technology Fund did all of their great work this last year with less than a $1,000 in donations. I want that to change. In speaking with Chris I know he has big plans for what the RTF can do with more money. If you are still reading please consider donating to the RTF this year. Their work is important and helps a lot of kids.

Not everyone has the resources to be able to donate and that is certainly understandable as well. If not, maybe consider talking to your employer about having a philanthropy day where you can give back to your community. I work at FireEye and am very grateful they adopted the Mandiant policy of having such a day where employees can donate their talents or time to help others. Last year I worked with my wife on creating curriculum for teaching programming to kids. This year we hope to go further with that goal and speak with kids here in the local high school about the infosec field. I personally consider myself fortunate to be in this field at this point in time. As others have said before me, to whom much is given much is expected. 

What the ER Can Learn from a Good CIRT

The following is a guest post I wrote for my friend and colleague, Chris Sanders on one of his blogs, appliednsm.com on July 1, 2015. 

--

I read Chris Sanders' recent blog post on investigations and prospective data collection with great interest. Before I explain why this is I should reveal some of my biases. I think transparency is something that is always important in understanding what place or frame of mind someone is coming from and we all have biases. First, Chris is my boss, a mentor and someone I consider a friend. In fairness to the reader just something I think they should know. Second, I was an emergency medical technician (EMT) in college and volunteered several hours a week on a very active rescue squad ambulance. This is not to confuse me with an ER doctor. I went on calls ranging from stabbings to motor vehicle accidents to people in cardiac arrest. In this way I have a different perception of emergency medicine than most people. Finally, I have worked as a security analyst in one form or another for several years now. Doing a bit with everything from intel to incident response to thinking a lot about triage in a console. These biases help me form the following opinions. Chris makes a very good argument for what he calls prospective collection and how to better apply a medical practitioner's approach to what we do as security analysts. I think he is on to something and the blog post made think in great detail about something I went through recently where I felt the emergency medical system could learn a lot from a SOC. While Chris focused more broadly on medical care I want to focus on emergency medical care.

My story really starts just over six weeks ago. I am a heavy sleeper so it was very strange to wake up around 6AM on a Sunday with difficulty breathing. Not just the stuffy nose kind of thing but the panic inducing feeling that my throat was closing on me. I tried to take some ibuprofen but was unable to swallow because of the level of constriction. All I could wonder was if I had been stung by something in my sleep and had developed an anaphylactic reaction. I live about 35 minutes away from a pretty good hospital so my wife and I got in the car and headed to the ER. On the way my ability to speak continued to worsen and my voice had physically changed such that it sounded like I had something in my throat. Upon arrival we got in line at the first check in station. I was now gasping for air like Lando Calirisian when Chewie was choking him out in the Empire Strikes Back. For some reason I started to think about myself as an alert and how a SOC would handle something like this. It was immediately outrageous to me that while I was struggling with breathing there were people in front of me with a sore arm and another with an upset stomach. I had to wait while insurance information was collected and people were asked why they were there. By the time I got to the desk my wife was speaking for me and I was given paperwork to fill out and told to wait. It occurred to me this was my first interaction with the hospital and they weren't doing any sort of triage at this point. Just collecting payment information. In the infosec world if you have a queue of alerts and you approach them serially you are doing something wrong. A high fidelity exfil alert should generally take precedence over an alert for a policy violation. As a responder I want to start distinguishing between priorities as soon as possible. This particular ER could improve here.

I then was able to make my way to the actual triage nurse after the people that were in front of me. I was asked a couple of questions and I conveyed that I was having difficulty breathing but still had an open airway. I was asked if I could swallow. I said I could swallow water but not food. I was also asked if I had been sick. I told the nurse I had a sore throat for the past three days. This is where I believe my experience as an EMT and the nurse's own biases started to work against me. Having been an EMT, even though I was panicked about my airway, I kept trying to remain calm. I think the nurse mistook this calm for "this is no big deal". I also think he immediately latched on to the sore throat being related to whatever was effecting me, thus resulting in less concern. These are mistakes I see many junior analysts make as well as managers with no experience in the trenches. The mistake of judging a situation based on people's reactions rather than the evidence at hand. I told the nurse calmly that I felt like my airway was closing and I was losing my ability to breath. As an EMT treating this it would immediately get my attention regardless of the patient's level of calmness. An inability to breath is a serious problem, just like one of those handful of alerts seasoned analysts see and immediately get a bad feeling about because it almost always means you have caught an actor mid-exfil or something of that level. New analysts are prone to not worry about more serious alerts and instead be more concerned on less worrisome ones where they have been conditioned to be upset by the environment. An example of this might be a concerned user contacting the CIRT because they noticed an unfamiliar icon on their desktop and are convinced based on a news report that this must have been maliciously placed there by an attacker. They happen to speak to a junior analyst or manager with no experience who hears the user's panic. They in turn panic because this must be really serious or why else would this user be so upset. Finally, an investigation reveals that an admin had installed Chrome for the user the day before trying to be helpful while the user was out. People are bad judges of lots of things. Let evidence lead you, not emotion. Good analysts know this and I wish the triage nurse in my experience also did.

Finally, I was in a hospital bed where another nurse (after another 45 minutes, I mean I only had a sore throat, right?) administered steroids via inhaler and liquid ibuprofen drip to help any swelling in the throat. I started to feel like I could breath again normally within a few minutes and then I waited and waited ...and waited for a doctor to show up. After about two hours a resident (this is a teaching hospital) came in to see me. I told her I was feeling better now but did not know what the root cause was. I was asked a surprisingly low number of questions. Not where had I been or what had I been doing recently? "Eat anything strange?", I was asked. The problem with this is you are relying on the patient to tell you what is strange. That is tantamount to asking a user in the HR department with a compromised system if they noticed any anomalous CPU usage recently. The doctors need to find out everything I ate recently and tell me if it is strange. I had a cantaloupe late the night before and later learned that can cause anaphylaxis in rare cases but I didn't mention it because I was unaware that might be a strange thing to eat. I knew shellfish could cause anaphylaxis and I had eaten some two days prior so I mentioned that even though it seemed to me to be too far away to be causing my current condition. The resident immediately latched onto this and assured me I probably had a small piece of shell stuck in my throat that had probably caused the whole episode but as I was feeling fine now there wasn't much else to do. This immediately reminded me of a junior analyst. Find something that might fit the description of what could have occurred and call it a day. As an analyst you want to be sure you have identified and then collected all of the relevant data sources that can help you come to a proper conclusion. In this case, the resident failed to adequately collect information and then seized on the first possibility. It also did not go unnoticed by me that by treating me with steroids and ibuprofen drip before ever looking in my throat the ER had committed the cardinal sin in a SOC of just running an AV scan on a system that is acting suspiciously enough that further investigation is warranted. It would now be much harder to know for sure what had been going on in my throat.

At about this time the supervising ER doctor came in. I immediately noticed more probing questions being asked. Additionally, this doctor wanted to see my throat so I got endoscoped (scope through the nostril down to the top of the larynx) twice. Why twice? Because the resident wanted a turn as she had never done it before. It was as enjoyable as it sounds. I am a sucker for learning experiences. The supervising doctor of course saw no problem with my airway as I had already been treated but did notice some white areas around the base of my tongue he was concerned about. He ultimately suggested a diagnosis of cancer before referring me to an ear nose and throat (ENT) doctor to check it out. After a couple of weeks of panic the ENT was able to see me and saw nothing he considered abnormal. The specialist asked even further probing questions and in the end thinks that as I had mowed a two acre hay field the night before my original episode that I had experienced an extreme reaction to a pollen allergy. While this was a relief to hear this was information neither the resident in the ER nor the ER doctor ever got because they did not ask enough questions. Ultimately, my impression from the final ER doctor was that a diagnosis was needed so he saw something strange in an area of which he was not an expert and thought it was the worst thing. While maybe being great at emergency medicine it seems as though he was acting as a junior analyst in his capacity to examine my throat. This is understandable. If you are unsure of what is going on though whether it be in a medical setting or in an active IR, suggesting it is the worst thing imaginable to concerned parties is probably not a responsible decision.

In summary, I understand this same situation could have played out differently at a different ER but I bet it also could have gone similarly at a lot of different ERs. Emergency medicine can at least be reminded by a good CIRT on the importance of collecting appropriate data, letting data lead the investigation, following up on those loose strings to pull, remaining calm, being honest about your assessment abilities in a specialized investigation, and finally getting the triage process better up front. It is an emergency after all.