Study sheet: System Security Fundamentals

📋 Course Outline

1. Security Objectives and Properties
2. Control Hijacking and Buffer Overflows
3. Integer Overflow and Browser Protections
4. Sandboxing, Isolation, and Robustness
5. Program Analysis Methods
6. Privileges and Access Control
7. Operating System Protection
8. Exploitation Techniques
9. Fuzzing Methods and Applications
10. Security Defenses and Terminology

📖 1. Security Objectives and Properties

🔑 Key Concepts & Definitions

• System Security : protects computer systems, operating systems, applications, networks, and data from unauthorized access, modification, destruction, data theft, malware, vulnerability exploitation, and denial of service

📝 Essential Points

• The CIA Triad consists of:
  • confidentiality
  • integrity
  • availability

📌 Authentication verifies who a user is, authorization determines what an authenticated user may do, and accountability tracks actions performed by users.

💡 Memory Hook

CIA: Confidentiality, Integrity, Availability

📖 2. Control Hijacking and Buffer Overflows

🔑 Key Concepts & Definitions

• Control Hijacking : manipulates a program's execution flow so that it executes unintended instructions or attacker-controlled code
• Buffer Overflow : occurs when a program writes more data into a fixed-size memory buffer than the buffer can hold, potentially overwriting adjacent memory

★ Must-know

• 🔄 A stack-based overflow can proceed as follows:
  1. write beyond the local buffer
  2. overwrite nearby stack data
  3. overwrite the return address
  4. return to an unintended location

Further detail

• Control hijacking commonly targets:

  • return addresses
  • function pointers
  • jump addresses
  • exception handlers
  • stack control information

• Common causes and historically unsafe functions include:

  • lack of bounds checking
  • unsafe string operations
  • incorrect memory management
  • improper input validation
  • programming errors
  • gets()
  • strcpy()
  • strcat()
  • sprintf()

💡 Memory Hook

Overflow → corrupted control data → unintended execution

📖 3. Integer Overflow and Browser Protections

★ Must-know

📐 Formula — An unsigned 8-bit integer represents values from 0 to 255, so $255 + 1 \rightarrow 0$ through wraparound; a 32-bit signed integer ranges from $-2^{31}$ to $2^{31}-1$.

• Integer overflow can turn a large calculated size into an unexpectedly small value, causing insufficient memory allocation and potentially enabling a buffer overflow.

• Browser protections include:

  • sandboxing
  • ASLR
  • DEP/NX
  • site isolation
  • process isolation
  • Same-Origin Policy

Further detail

• Integer overflow can affect:
  • memory allocation
  • array sizes
  • buffer sizes
  • loop conditions
  • file sizes
  • length calculations

💡 Memory Hook

Input → arithmetic overflow → incorrect size → memory corruption

📖 4. Sandboxing, Isolation, and Robustness

🔑 Key Concepts & Definitions

• Sandboxing : executes a program inside a restricted environment with limited access to files, network resources, hardware, processes, system calls, and sensitive data
• Robust Software : continues to behave safely and correctly when it receives unexpected input or encounters abnormal conditions

📝 Essential Points

📌 Sandboxing restricts what an application can do through policies and limited permissions, whereas isolation separates processes, users, applications, or resources to prevent unwanted interaction.

• Important robustness techniques include:
  • input validation
  • output encoding
  • safe error handling
  • least privilege
  • secure memory management
  • secure defaults
  • code review
  • security testing

💡 Memory Hook

Sandboxing restricts actions, whereas isolation separates components

📖 5. Program Analysis Methods

★ Must-know

📌 Static analysis examines a program without executing it, dynamic analysis examines it during execution, and concolic analysis combines concrete execution with symbolic reasoning.

• Concolic execution runs a program with concrete input while maintaining symbolic constraints, then generates new inputs to explore alternative paths; its main limitation is path explosion.

Further detail

• Static analysis can detect:

  • possible buffer overflows
  • uninitialized variables
  • dead code
  • memory problems
  • dangerous function usage
  • certain injection vulnerabilities
  • incorrect data flows

• Dynamic analysis can detect memory violations, crashes, invalid memory accesses, runtime behavior, and resource problems, but it only explores executed paths.

💡 Memory Hook

Static sees, dynamic does, concolic combines

📖 6. Privileges and Access Control

🔑 Key Concepts & Definitions

• Privilege : permission granted to a user, process, or program to perform a particular operation such as reading a file, writing a file, executing a program, accessing a device, or changing system configuration
• Access Control : determines which subjects can access which resources and which operations they are allowed to perform

📝 Essential Points

📌 Vertical privilege escalation moves from a low-privileged user to a high-privileged user, whereas horizontal privilege escalation gives one user access to another user at a similar privilege level.

📌 Authentication answers “Who are you?”, whereas authorization answers “What are you allowed to do?” after authentication.

• DAC lets resource owners decide access, MAC enforces centrally defined policies and labels, RBAC assigns permissions to roles, and ABAC bases decisions on attributes such as user, role, device, location, resource, time, and environment.

💡 Memory Hook

Authentication asks who, authorization asks what

📖 7. Operating System Protection

🔑 Key Concepts & Definitions

• Process Isolation : gives each process its own protected address space so that one process cannot normally access another process's memory directly

★ Must-know

• Operating system security protects:

  • processes
  • memory
  • files
  • users
  • devices
  • system calls
  • the kernel
  • network resources

• Memory pages can have read, write, and execute permissions, represented by R, W, and X; an R-X page is readable and executable but not writable.

Further detail

• Major operating-system security mechanisms include:
  • authentication
  • authorization
  • access control
  • process isolation
  • memory protection
  • privilege separation
  • secure boot
  • file permissions
  • auditing and logging
  • security updates

📖 8. Exploitation Techniques

🔑 Key Concepts & Definitions

• Exploitation : the process of taking advantage of a software or system vulnerability to cause unintended behavior
• Code Injection : occurs when an attacker causes unintended code or commands to be interpreted or executed, as in SQL injection, command injection, or script injection
• Use-After-Free : occurs when a program uses memory after that memory has been released, potentially causing crashes, data corruption, unexpected behavior, or code-execution vulnerabilities
• Return-Oriented Programming : uses existing instruction sequences called gadgets to perform desired behavior without injecting an entire new program

💡 Memory Hook

Vulnerability → analysis → trigger → unintended behavior → impact

📖 9. Fuzzing Methods and Applications

🔑 Key Concepts & Definitions

• Fuzzing : an automated testing technique that supplies large amounts of unexpected, malformed, random, or specially generated input to a program and monitors its behavior

★ Must-know

• 🔄 A basic fuzzing process:
  1. generates test input
  2. submits it to an application
  3. monitors for crashes or interesting behavior
  4. saves notable inputs
  5. analyzes the results

📌 Mutation-based fuzzing modifies valid inputs, generation-based fuzzing creates inputs from a specification or grammar, black-box fuzzing has little internal knowledge, white-box fuzzing uses internal structure, and grey-box fuzzing uses limited feedback such as code coverage.

Further detail

• Fuzzing can discover:
  • crashes
  • buffer overflows
  • integer-related bugs
  • memory corruption
  • parser vulnerabilities
  • unexpected application states
  • input-validation errors

💡 Memory Hook

Generate → execute → monitor → save → analyze

📖 10. Security Defenses and Terminology

★ Must-know

• ASLR randomizes memory locations, DEP/NX prevents execution from designated data regions, stack canaries detect certain stack overflows, CFI restricts execution to valid control-flow paths, sandboxing restricts compromised applications, and least privilege minimizes permissions.

📌 A vulnerability is a weakness in a system, an exploit is a technique, code, or input that takes advantage of the weakness, and an attack is broader malicious activity using vulnerabilities or exploits to achieve a goal.

Further detail

• The principal program-analysis approaches are static, dynamic, and concolic analysis, followed by fuzzing to find unusual inputs, bug analysis, and fixing.

📌 Fuzzing is automated input testing focused on unexpected behavior, whereas penetration testing is a broader, usually scoped security assessment that investigates complete attack paths.

💡 Memory Hook

Vulnerability → exploit → attack → impact

📊 Synthesis Tables

Program Analysis Comparison

Feature	Static	Dynamic	Concolic
Program execution	No	Yes	Yes
Actual input	Not necessarily	Yes	Yes
Symbolic constraints	Usually no	Usually no	Yes
Main limitation	False positives	Missed paths	Path explosion

Access Control Models

Model	Decision basis	Typical characteristic
DAC	Resource owner	Owner grants access
MAC	Central policy and labels	Strictly controlled environments
RBAC	Roles	Permissions assigned to roles
ABAC	Attributes and context	Uses user, device, location, time, and environment