Troubleshooting with CheckAsm — Common FixesCheckAsm is a static-analysis and validation tool for assembly code that helps catch syntax errors, undefined labels, incorrect instruction formats, and platform-specific calling convention issues. This guide walks through common problems you may encounter when using CheckAsm, how to interpret its messages, and practical fixes to get your assembly code building and running correctly.
1. Understanding CheckAsm output
CheckAsm reports errors and warnings with line numbers, error codes (if enabled), and short diagnostic messages. Typical outputs include:
- Syntax errors (unexpected tokens, missing operands)
- Undefined symbols or labels
- Instruction operand size mismatches
- Invalid addressing modes for the target architecture
- Calling convention or ABI violations
- Performance or safety warnings (e.g., unaligned memory access)
Tip: Start by fixing errors marked as “error” before addressing “warning” messages; errors usually prevent successful assembly.
2. Common syntax errors and fixes
Symptoms: CheckAsm reports “unexpected token”, “missing operand”, or “unterminated string” on a line.
Fixes:
- Verify instruction spelling and case sensitivity. Some assemblers are case-sensitive for directives or labels.
- Ensure correct operand count and separators (commas between operands).
- Check for stray characters (tabs vs. spaces, non-ASCII characters copied from documentation).
- For string literals, ensure matching quotes and escape sequences are valid for the assembler.
Example: If CheckAsm flags:
mov eax ebx
Fix by adding the missing comma:
mov eax, ebx
3. Undefined labels and symbol resolution
Symptoms: “Undefined symbol: label” or “relocation failed: unknown symbol”.
Fixes:
- Confirm label names match exactly where defined and referenced (watch for typos).
- Ensure labels are within the proper scope (local vs. global). Prefix local labels with the required character (e.g., .L or @ depending on assembler).
- If using multiple source files, ensure the symbol is exported (global) in the defining file and declared extern in the using file if required.
- Check for conditional assembly blocks that may omit label definitions under certain build flags.
4. Operand size and register mismatches
Symptoms: “operand size mismatch”, “invalid register for instruction”.
Fixes:
- Make sure immediate values fit the required size (byte, word, dword, etc.). Use explicit size modifiers (e.g., byte ptr, word ptr) if needed.
- Use correct-sized registers for the instruction (e.g., use al/ax/eax/rax appropriately).
- For x86/x86-64, ensure 64-bit instructions use 64-bit registers and operand-size prefixes when necessary.
Example:
mov rax, ebx ; wrong — mixing 64-bit and 32-bit registers
Correct:
mov rax, rbx
mov eax, ebx
5. Invalid addressing modes and memory operands
Symptoms: “invalid memory operand” or “unsupported addressing mode”.
Fixes:
- Match addressing modes to target architecture syntax. For instance, AT&T syntax differs from Intel: AT&T uses % and source/destination order is swapped.
- Check scale-index-base addressing correctness: base + index*scale + displacement.
- Ensure proper use of segment registers or far pointers on segmented architectures.
Example (Intel):
mov eax, [ebx + ecx*4 + 8]
6. Calling convention and ABI violations
Symptoms: Runtime crashes, stack corruption, or CheckAsm warnings about clobbered registers.
Fixes:
- Verify function prologues/epilogues follow the platform ABI (registers saved/restored, stack alignment).
- Ensure caller-saved vs callee-saved registers are respected.
- On x86-64 System V, keep stack 16-byte aligned before calls. On Windows x64, shadow space must be reserved.
- Correctly declare extern and global symbols for inter-language calls.
Quick checklist:
- Push/pop pairs balanced
- RSP alignment conserved across call boundaries
- Return values placed in the ABI-specified register(s)
7. Relocations and position-independent code (PIC)
Symptoms: “relocation truncated”, “absolute address not allowed in PIC”.
Fixes:
- Use RIP-relative addressing on x86-64 for position-independent code.
- For global data access in PIC, load addresses via GOT/PLT when required.
- Avoid embedding absolute addresses; use assembler/linker-provided relocations or labels.
8. Cross-assembly and target mismatch issues
Symptoms: Instructions accepted but runtime behavior incorrect, or CheckAsm reports “unknown instruction for target”.
Fixes:
- Ensure CheckAsm target architecture/CPU flags match the intended platform (e.g., armv7 vs armv8, x86 vs x86-64).
- Enable the correct instruction set extensions (SSE, AVX, NEON) in CheckAsm settings if using extended instructions.
- Assemble with the same assembler syntax expected by your toolchain (Intel vs AT&T).
9. Macro and preprocessor problems
Symptoms: Expanded code contains errors, or macros behave inconsistently.
Fixes:
- Inspect macro expansions (many assemblers provide a way to dump expanded code) to see the exact emitted lines CheckAsm is analyzing.
- Guard macro arguments with parentheses and appropriate operand-size specifiers.
- Avoid using local labels inside macros, or ensure they expand uniquely (use assembler local-label syntax if available).
10. Linker-related complaints and symbol visibility
Symptoms: Linker errors after assembly: undefined references, multiple definition errors.
Fixes:
- Check symbol visibility and use of .global/.globl/.export directives correctly.
- Avoid duplicating global symbols across object files; make variables static/local where appropriate.
- For data sections, ensure correct section qualifiers (.data vs .rodata) and alignment directives.
11. Performance and safety warnings
Symptoms: CheckAsm warns about unaligned access, deprecated instructions, or potential data races.
Fixes:
- Align data structures and stack appropriately for the architecture.
- Replace deprecated or unsafe instructions with recommended alternatives.
- For concurrency issues, ensure atomic operations or proper synchronization primitives are used.
12. Using CheckAsm features to help debugging
- Enable verbose diagnostics to get fuller context for errors.
- Use a source map or dialog option to show expanded macro lines.
- Turn on target-specific checks to catch ABI/ISA mismatches early.
- Generate an annotated listing to correlate machine code offsets with source lines.
13. Example troubleshooting session
Problem: runtime crash after calling an assembly routine from C.
Steps:
- Reproduce with a small test case.
- Run CheckAsm and fix any assembly errors/warnings.
- Verify calling convention: register usage, stack alignment, preserved registers.
- Inspect generated object with objdump/disassembler to ensure instructions match expectations.
- Run under a debugger to check the return address, stack pointer, and register values at crash time.
14. When to seek help
- If diagnostics are unclear, collect: CheckAsm output, minimal reproducible assembly snippet, target architecture and assembler syntax, and any linker flags.
- Share the smallest test case that reproduces the issue; this makes it much faster to diagnose.
Troubleshooting assembly with CheckAsm is iterative: fix high-priority errors, re-run, and verify runtime behavior. Focusing on ABI correctness, operand sizes, and label resolution usually resolves the majority of issues.