Malware Development Series: Part 3

⚠️ Ethical Use Disclaimer: This content is for educational and ethical security research purposes only. Knowledge of malware development should only be used for proper penetration testing, red team operations and defensive security research. Unauthorized use will be subject to criminal prosecution.

Introduction

In Part 3 of the Malware Development Series, we examine how modern security solutions detect malware. We provide a introductory account of Windows processes and threads, memory types, and the Process Environment Block (PEB). An understanding of these concepts is essential for security professionals, both offensive and defensive, wanting to develop or analyze advanced malware.

Detection Mechanisms

Security products use many methods to successfully identify and target malware. Knowing these methods is important for both evasion and identification. In the previous Parts, we introduced the concepts and fundamentals of Windows architecture and memory management; now we can study how some of these core systems implement detection capabilities.

Static/Signature Detection

Signature detection works by finding identifiable byte patterns, string, or attributes within a file. Security vendors use tools such as YARA to create rules that detect known malware. One example of a signature could be a byte pattern that corresponds to a specific shellcode or a hardcoded string. Signature detection is an easy way to identify known threats, but it can be easily avoided by changing the code or obfuscating unique values.

Tip: Avoid hardcoding values and dynamically generate or retrieve them at runtime to evade static signatures.

Hashing Detection

Hash-based detection compares the hash (e.g., MD5, SHA256) of a file against a database of known malware hashes. This method is fast but trivial to bypass—changing even a single byte alters the hash and evades detection. However, it is often used in combination with other techniques.

Heuristic Detection

Heuristic analysis aims to detect new or modified malware by looking for suspicious characteristics or behaviors.

• Static Heuristic Analysis: Decompiles and analyzes code for similarities to known malware. If a certain percentage matches, the file is flagged.
• Dynamic Heuristic Analysis (Sandboxing): Executes the file in a controlled environment and observes its behavior. Sequences like memory allocation, downloading shellcode, and executing it are considered highly suspicious.

Malware often includes anti-sandboxing techniques to detect when it is running in a virtual environment and will alter its behavior to appear benign.

Behavior-Based Detection

Behavior-based detection involves monitoring running processes to see if they perform any suspicious actions like loading DLLS, calling sensitive APIs, or connecting to the internet. If the behavior is suspicious, the process may be killed or be scanned in-memory. Evasion techniques include pretending to be a benign process and encrypting one or more memory regions to prohibit the detection.

API Hooking

API hooking is implemented by Endpoint Detection and Response (EDR) products to inspect calls to abused Windows APIs. They also analyze the parameters in real-time to find malicious behavior, even after obfuscation. API hook evasion can be done through DLL unhooking, as well as direct syscalls, which will be discussed in later posts.

Side Note: Direct Syscalls: Bypasses API hooks by invoking system calls directly using syscall numbers. This technique is harder to detect but depends on syscall number stability across Windows builds

IAT Checking

The Import Address Table (IAT) lists all external functions and libraries used by a PE file. Security solutions analyze the IAT for suspicious imports (e.g., cryptographic or file management functions in ransomware). Tools like dumpbin.exe can be used to inspect the IAT. Advanced malware may use API hashing or dynamic resolution to evade IAT-based detection.

Figure: Import Address Table (IAT) structure showing how external functions are resolved.

Manual Analysis

Even if automated detection is bypassed, skilled analysts can reverse engineer and identify malware through manual analysis. Anti-reversing techniques, such as debugger and virtualization detection, can slow down analysis but are not foolproof.

🔍 Evasion Techniques: Explore common evasion methods in the Evasion Techniques tab of our interactive visualization. Learn how techniques like polymorphism, packing, and anti-debugging target specific detection layers.

Windows Processes and Threads

To understand the inner structure of Windows processes is crucial for malware development as well as analysis. Ideas in this section connect with the detection methods above as many security solutions monitor process behavior, memory allocation and the activity of threads.

💡 Practical Application: The exercises in this section will help you build tools to analyze these structures hands-on, giving you practical experience with the concepts used by both malware authors and security researchers.

What is a Windows Process?

A Windows process is an instance of a running program, created by the system or a user. Each process consumes resources such as memory, disk, and CPU time. Processes are isolated from each other for security and stability.

Process Threads

A process contains one or more threads, which are independent sequences of execution. Threads share the process's resources but can execute concurrently. The operating system schedules threads and manages their execution context.

Process Memory Types

Building on the virtual memory concepts covered in Part 2, Windows processes utilize different types of memory regions:

• Private Memory: Dedicated to a single process, used for process-specific data.
• Mapped Memory: Shared between processes, used for shared libraries or inter-process communication.
• Image Memory: Contains the code and data of executables and loaded DLLs.

Understanding these memory types is crucial for malware that needs to inject code, share data between processes, or analyze running applications.

Process Environment Block (PEB)

The PEB is a data structure used by Windows; it holds details about a process, for example startup values, loaded dlls and heap details. Each process has its own PEB, it can be used by the OS and is also a target for malware to further its stealthy operations. This is a logical extension of the Windows architecture principles we discussed in Part 1.

Key PEB Members:

• BeingDebugged: Indicates if the process is being debugged.
• Ldr: Points to a structure listing loaded modules (DLLs).
• ProcessParameters: contains process parameter information such as the command line.
• SessionId: Identifies the Terminal Services session identifier associated with the current process.

Figure: Process Environment Block (PEB) structure containing process information and loaded modules.

Malware can use the PEB to locate loaded modules, spoof command-line arguments, or detect debuggers.

Thread Environment Block (TEB)

The TEB is a per-thread data structure that contains information about the thread's environment, security context and a pointer to the PEB for the thread's owning process as well as allocated Thread Local Storage (TLS) slots for thread data.

Side Note: TLS Callbacks: Functions executed before the main entry point, often used by malware to evade static analysis or inject early logic.

TEB Analysis with WinDbg

Here's practical TEB analysis output from WinDbg showing real structure data:

Figure: WinDbg !teb command showing Thread Environment Block information including stack boundaries, client IDs, and PEB address.

This data is useful for:

• Stack analysis: StackBase/StackLimit help locate stack boundaries for shellcode injection
• Thread identification: ClientId provides Process ID (0x1c64) and Thread ID (0x2ce0) for targeting
• Anti-debugging: LastErrorValue can reveal debugging artifacts
• PEB location: Direct access to PEB structure at 0x00000042f6a62000

PEB Structure Analysis with WinDbg

For detailed PEB structure examination:

Figure: WinDbg dt _PEB command showing detailed Process Environment Block structure with debugging flags, image base, and loader information.

TEB Structure Analysis

For complete TEB structure analysis:

Figure: WinDbg dt _TEB command showing comprehensive Thread Environment Block structure including NT_TIB, client information, and GDI data.

This data is useful for:

• Anti-debugging: BeingDebugged flag (0x1) immediately reveals debugging presence
• Module enumeration: Ldr pointer (0x00007ffaef1f39c0) enables direct API resolution
• Process injection: ImageBaseAddress (0x00007ff7f8790000) provides target for process hollowing
• Forensic analysis: ProcessParameters reveal execution context
• Hook bypassing: Module lists allow direct syscall resolution without API hooks

Side Note: Malware can use the Ldr list in PEB to locate kernel32.dll, parse its export table manually, and resolve APIs without calling LoadLibrary or GetProcAddress, bypassing IAT detection.

Figure: Thread Environment Block (TEB) structure diagram showing the relationship between TEB fields and process/thread management.

Process and Thread Handles

Each process and thread has a unique identifier (PID and TID). Handles to these objects can be obtained using APIs like OpenProcess and OpenThread, which are required for actions such as suspending, resuming, or terminating processes and threads. Always close handles with CloseHandle to avoid resource leaks.

Practical Exercises

To reinforce the concepts covered in this part, complete the following hands-on exercises.

Exercise 1: PEB Analysis Tool

Objective: Create a simple tool to analyze the Process Environment Block (PEB) of a running process.

Requirements:

• Visual Studio or your preferred C/C++ development environment
• Basic understanding of Windows data structures

Learning Objectives:

• Understand PEB structure access
• Explore module enumeration techniques

Exercise 2: IAT Inspector

Objective: Build a tool to analyze the Import Address Table (IAT) of a PE file.

Learning Objectives:

• Understand PE file structure
• Practice IAT parsing techniques
• Identify suspicious API imports

Exercise 4: Memory Analysis Tool

Objective: Create a tool to analyze memory regions of a target process.

Learning Objectives:

• Understand memory region analysis
• Practice process memory enumeration
• Learn about memory protection flags
• Explore different memory types

Exercise Completion Checklist

• Exercise 1: Successfully compile and run PEB analysis tool
• Exercise 2: Parse IAT from a sample PE file (try with notepad.exe)
• Exercise 3: Analyze memory of a running process

Additional Challenge

Advanced Exercise: Combine concepts from all exercises to create a process analysis tool that:

1. Analyzes the target process's PEB
2. Enumerates loaded modules and their IATs
3. Implements anti-debugging checks
4. Maps memory regions and identifies suspicious sections

This exercise integrates the Windows architecture knowledge from Part 1, memory management concepts from Part 2, and detection mechanisms from this part.

Conclusion

This section examined the primary detection mechanisms utilized by security solutions and introduced the basics of Windows processes, threads, types of memory, and the PEB. These ideas build directly on the Windows architecture and memory management concepts discussed in Part 1 and Part 2, respectively.

Next in Series: Part 4 will dive deeper into anti-analysis and anti-reversing techniques, including debugger detection, virtualization checks, and advanced evasion strategies.

Additional Resources

Detection Mechanisms & Analysis:

• YARA Documentation - Pattern matching engine for malware detection
• Cuckoo Sandbox - Dynamic malware analysis platform
• PEiD - PE file analysis and packer detection
• Detect It Easy (DIE) - Program for determining types of files
• Process Hacker - Advanced process monitoring tool

Windows Internals & Debugging:

• Windows Internals by Pavel Yosifovich et al. - Comprehensive Windows architecture guide
• Microsoft Docs: Windows Processes and Threads - Official Windows process documentation
• WinDbg Documentation - Complete WinDbg debugging reference
• Windows Internals Book Series - Deep dive into Windows architecture

PEB/TEB & Memory Analysis:

• Undocumented Windows Structures - Internal Windows data structures
• Process Environment Block (PEB) Analysis - Detailed PEB structure documentation
• Thread Environment Block (TEB) Reference - Comprehensive TEB analysis
• Windows Memory Management - Official memory management documentation

Malware Analysis & Research:

• Practical Malware Analysis by Michael Sikorski - Essential malware analysis textbook
• The Malware Analyst's Cookbook - Advanced analysis techniques
• Malware Traffic Analysis - Real-world malware samples and analysis
• SANS Reverse Engineering Malware - Professional training course

API Hooking & EDR Bypass:

• Detours Library Documentation - Microsoft's API hooking library
• API Monitor - Real-time API monitoring tool
• Sysmon Configuration - Advanced system monitoring
• EDR Bypass Techniques - Workshop on Windows API and EDR evasion

Important Legal and Ethical Considerations

Utilize knowledge for authorized testing, obtain written authorization, disclose responsibly, maintain operational security, and keep a record of what you do for legal protection.

Takeaway: The reason you are learning malware development is to make good legitimate security better and not cause harm. It is your responsibility to ensure that what you are doing is legal, authorized, and appropriately conducted.