When Things Go Wrong

Security incidents happen to everyone-breaches, malware, data leaks. What matters is how you respond.

Fast, effective response minimizes damage. Panic and improvisation make it worse.

---

The Incident Response Lifecycle

---

Phase 1: Preparation

Before incidents happen:

Build Your Team

Role	Responsibility
Incident Commander	Decisions, communication
Technical Lead	Investigation, containment
Communications	Internal/external messaging
Legal	Compliance, notification

Create Runbooks

Pre-written playbooks for common scenarios:

• Malware infection
• Data breach
• DDoS attack
• Account compromise

Prepare Tools

• Log aggregation (ELK, Splunk)
• Forensic tools
• Communication channels (out-of-band)
• Contact lists

---

Phase 2: Detection

Recognize that an incident is occurring:

Detection Sources

Source	Example
Monitoring alerts	Unusual login patterns
User reports	”My computer is acting weird”
External notification	Researcher, customer, attacker
Log analysis	Failed auth spikes

Triage Questions

1. What is happening?
2. When did it start?
3. What systems are affected?
4. Is it ongoing?
5. What’s the potential impact?

---

Phase 3: Containment

Stop the bleeding:

Short-Term Containment

Action	Purpose
Isolate system	Prevent lateral movement
Block IPs	Stop ongoing attack
Disable accounts	Prevent access
Preserve evidence	Don’t destroy logs

# Example: Isolate network
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP

# Or network-level
# Move to quarantine VLAN

Long-Term Containment

Keep business running while you investigate:

• Temporary workarounds
• Clean systems in parallel
• Monitor for re-infection

---

Phase 4: Eradication

Remove the threat completely:

Find Root Cause

• How did they get in?
• What did they access?
• What did they leave behind?

Clean Up

• Remove malware
• Close vulnerabilities
• Reset compromised credentials
• Patch exploited systems

---

Phase 5: Recovery

Return to normal operations:

Restore Safely

1. Verify system is clean
2. Restore from known-good backup
3. Monitor closely after restoration
4. Gradual return to production

Validation

• Systems functioning correctly
• Security controls in place
• No signs of persistent access

---

Phase 6: Lessons Learned

Every incident is a learning opportunity.

Post-Mortem Meeting

Within 1-2 weeks:

• What happened (timeline)
• What went well
• What could improve
• Action items

Document Everything

# Incident Report: [Title]

## Summary
Brief description of what happened.

## Timeline
- HH:MM - Detection
- HH:MM - Containment began
- HH:MM - Root cause identified
- HH:MM - Systems restored

## Impact
- Systems affected
- Data exposed
- Duration
- Cost

## Root Cause
How the incident occurred.

## Response Actions
What was done to contain and eradicate.

## Lessons Learned
What to improve.

## Action Items
- [ ] Item 1 (Owner, Due date)
- [ ] Item 2 (Owner, Due date)

---

Practice Exercises

Exercise 1: Triage (Beginner)

An employee reports: “I can’t access my email and there are weird files on my desktop.”

What are your first 3 questions?

Answer

1. When did you first notice this?
2. Did you click any links or open attachments recently?
3. Are your coworkers experiencing the same issue?

Exercise 2: Containment (Intermediate)

You’ve confirmed malware on a developer’s laptop that has SSH access to production servers.

What containment actions do you take?

Answer

1. Isolate the laptop from network
2. Revoke developer’s SSH keys
3. Check production access logs
4. Force password reset on affected accounts
5. Monitor for unusual production activity

Exercise 3: Post-Mortem (Advanced)

Write a brief post-mortem for this scenario:

• Attacker gained access via phishing
• Had access for 3 days before detection
• Exfiltrated customer database

---

Knowledge Check

1. What are the six phases of incident response?

2. Why preserve evidence during containment?

3. What’s the difference between short-term and long-term containment?

4. Why do a post-mortem?

5. What’s the first thing you should do when you detect an incident?

Answers

1. Preparation, Detection, Containment, Eradication, Recovery, Lessons Learned.

2. For investigation and potential legal action. Destroying evidence makes it impossible to understand what happened.

3. Short-term = immediate isolation. Long-term = temporary workarounds while you investigate and clean up properly.

4. Learn and improve. Understand what happened, what worked, what didn’t, so you’re better prepared next time.

5. Don’t panic. Then assess the scope and impact before taking containment actions.

---

Summary

Phase	Goal
Preparation	Be ready before incidents
Detection	Recognize incidents quickly
Containment	Stop the damage
Eradication	Remove the threat
Recovery	Return to normal
Lessons	Improve for next time

---

What’s Next?

🎯 Continue learning:

• Security Logging - Detection foundation
• Zero Trust - Reduce blast radius

---

You now know how to respond when security fails. 🚨