What the ER Can Learn from a Good CIRT

The following is a guest post I wrote for my friend and colleague, Chris Sanders on one of his blogs, appliednsm.com on July 1, 2015. 

--

I read Chris Sanders' recent blog post on investigations and prospective data collection with great interest. Before I explain why this is I should reveal some of my biases. I think transparency is something that is always important in understanding what place or frame of mind someone is coming from and we all have biases. First, Chris is my boss, a mentor and someone I consider a friend. In fairness to the reader just something I think they should know. Second, I was an emergency medical technician (EMT) in college and volunteered several hours a week on a very active rescue squad ambulance. This is not to confuse me with an ER doctor. I went on calls ranging from stabbings to motor vehicle accidents to people in cardiac arrest. In this way I have a different perception of emergency medicine than most people. Finally, I have worked as a security analyst in one form or another for several years now. Doing a bit with everything from intel to incident response to thinking a lot about triage in a console. These biases help me form the following opinions. Chris makes a very good argument for what he calls prospective collection and how to better apply a medical practitioner's approach to what we do as security analysts. I think he is on to something and the blog post made think in great detail about something I went through recently where I felt the emergency medical system could learn a lot from a SOC. While Chris focused more broadly on medical care I want to focus on emergency medical care.

My story really starts just over six weeks ago. I am a heavy sleeper so it was very strange to wake up around 6AM on a Sunday with difficulty breathing. Not just the stuffy nose kind of thing but the panic inducing feeling that my throat was closing on me. I tried to take some ibuprofen but was unable to swallow because of the level of constriction. All I could wonder was if I had been stung by something in my sleep and had developed an anaphylactic reaction. I live about 35 minutes away from a pretty good hospital so my wife and I got in the car and headed to the ER. On the way my ability to speak continued to worsen and my voice had physically changed such that it sounded like I had something in my throat. Upon arrival we got in line at the first check in station. I was now gasping for air like Lando Calirisian when Chewie was choking him out in the Empire Strikes Back. For some reason I started to think about myself as an alert and how a SOC would handle something like this. It was immediately outrageous to me that while I was struggling with breathing there were people in front of me with a sore arm and another with an upset stomach. I had to wait while insurance information was collected and people were asked why they were there. By the time I got to the desk my wife was speaking for me and I was given paperwork to fill out and told to wait. It occurred to me this was my first interaction with the hospital and they weren't doing any sort of triage at this point. Just collecting payment information. In the infosec world if you have a queue of alerts and you approach them serially you are doing something wrong. A high fidelity exfil alert should generally take precedence over an alert for a policy violation. As a responder I want to start distinguishing between priorities as soon as possible. This particular ER could improve here.

I then was able to make my way to the actual triage nurse after the people that were in front of me. I was asked a couple of questions and I conveyed that I was having difficulty breathing but still had an open airway. I was asked if I could swallow. I said I could swallow water but not food. I was also asked if I had been sick. I told the nurse I had a sore throat for the past three days. This is where I believe my experience as an EMT and the nurse's own biases started to work against me. Having been an EMT, even though I was panicked about my airway, I kept trying to remain calm. I think the nurse mistook this calm for "this is no big deal". I also think he immediately latched on to the sore throat being related to whatever was effecting me, thus resulting in less concern. These are mistakes I see many junior analysts make as well as managers with no experience in the trenches. The mistake of judging a situation based on people's reactions rather than the evidence at hand. I told the nurse calmly that I felt like my airway was closing and I was losing my ability to breath. As an EMT treating this it would immediately get my attention regardless of the patient's level of calmness. An inability to breath is a serious problem, just like one of those handful of alerts seasoned analysts see and immediately get a bad feeling about because it almost always means you have caught an actor mid-exfil or something of that level. New analysts are prone to not worry about more serious alerts and instead be more concerned on less worrisome ones where they have been conditioned to be upset by the environment. An example of this might be a concerned user contacting the CIRT because they noticed an unfamiliar icon on their desktop and are convinced based on a news report that this must have been maliciously placed there by an attacker. They happen to speak to a junior analyst or manager with no experience who hears the user's panic. They in turn panic because this must be really serious or why else would this user be so upset. Finally, an investigation reveals that an admin had installed Chrome for the user the day before trying to be helpful while the user was out. People are bad judges of lots of things. Let evidence lead you, not emotion. Good analysts know this and I wish the triage nurse in my experience also did.

Finally, I was in a hospital bed where another nurse (after another 45 minutes, I mean I only had a sore throat, right?) administered steroids via inhaler and liquid ibuprofen drip to help any swelling in the throat. I started to feel like I could breath again normally within a few minutes and then I waited and waited ...and waited for a doctor to show up. After about two hours a resident (this is a teaching hospital) came in to see me. I told her I was feeling better now but did not know what the root cause was. I was asked a surprisingly low number of questions. Not where had I been or what had I been doing recently? "Eat anything strange?", I was asked. The problem with this is you are relying on the patient to tell you what is strange. That is tantamount to asking a user in the HR department with a compromised system if they noticed any anomalous CPU usage recently. The doctors need to find out everything I ate recently and tell me if it is strange. I had a cantaloupe late the night before and later learned that can cause anaphylaxis in rare cases but I didn't mention it because I was unaware that might be a strange thing to eat. I knew shellfish could cause anaphylaxis and I had eaten some two days prior so I mentioned that even though it seemed to me to be too far away to be causing my current condition. The resident immediately latched onto this and assured me I probably had a small piece of shell stuck in my throat that had probably caused the whole episode but as I was feeling fine now there wasn't much else to do. This immediately reminded me of a junior analyst. Find something that might fit the description of what could have occurred and call it a day. As an analyst you want to be sure you have identified and then collected all of the relevant data sources that can help you come to a proper conclusion. In this case, the resident failed to adequately collect information and then seized on the first possibility. It also did not go unnoticed by me that by treating me with steroids and ibuprofen drip before ever looking in my throat the ER had committed the cardinal sin in a SOC of just running an AV scan on a system that is acting suspiciously enough that further investigation is warranted. It would now be much harder to know for sure what had been going on in my throat.

At about this time the supervising ER doctor came in. I immediately noticed more probing questions being asked. Additionally, this doctor wanted to see my throat so I got endoscoped (scope through the nostril down to the top of the larynx) twice. Why twice? Because the resident wanted a turn as she had never done it before. It was as enjoyable as it sounds. I am a sucker for learning experiences. The supervising doctor of course saw no problem with my airway as I had already been treated but did notice some white areas around the base of my tongue he was concerned about. He ultimately suggested a diagnosis of cancer before referring me to an ear nose and throat (ENT) doctor to check it out. After a couple of weeks of panic the ENT was able to see me and saw nothing he considered abnormal. The specialist asked even further probing questions and in the end thinks that as I had mowed a two acre hay field the night before my original episode that I had experienced an extreme reaction to a pollen allergy. While this was a relief to hear this was information neither the resident in the ER nor the ER doctor ever got because they did not ask enough questions. Ultimately, my impression from the final ER doctor was that a diagnosis was needed so he saw something strange in an area of which he was not an expert and thought it was the worst thing. While maybe being great at emergency medicine it seems as though he was acting as a junior analyst in his capacity to examine my throat. This is understandable. If you are unsure of what is going on though whether it be in a medical setting or in an active IR, suggesting it is the worst thing imaginable to concerned parties is probably not a responsible decision.

In summary, I understand this same situation could have played out differently at a different ER but I bet it also could have gone similarly at a lot of different ERs. Emergency medicine can at least be reminded by a good CIRT on the importance of collecting appropriate data, letting data lead the investigation, following up on those loose strings to pull, remaining calm, being honest about your assessment abilities in a specialized investigation, and finally getting the triage process better up front. It is an emergency after all.