Skip to content

Commit

Permalink
Add docs to install Python3.11 and setup venv.
Browse files Browse the repository at this point in the history
Add rzil-compiler as submodule

Add cmd option for rzil generation.

Add structs which hold IL ops, instructions and RZIL getter.

Generate `hexagon_il.h` with insn il op getter declarations.

Add reuse license info and ignore VSCode folder.

Use lists of insn names instead of objects.

Init rzil-compiler.

Get compiled instrucition RZIL behavior.

Generate IL op getter.

Add option to skip pcpp step

Set il_ops structs for Duplex instructions.

Save getter along with rzil behavior

Build il files only if flag says so

Use getter saved in il_ops

Remove set il_ops from Duplex Instructions.

Fill complete HexILInsn struct

Fix: Actual declarations into hexagon_il.h not just names.

Update rizin files.

Fix type related build errros.

update rzil-compiler

Add progress bar while parsing/compiling isntructions

Run black

Update rzil-compiler

Extract `set_il_op` for instruction to method.

Add class and isa_id to HexOp.

Set `isa_id` char in template.

Add ISA2REG function.

Add ISA2IMM

Refactor: Extract method to generate register enums.

Rewrite `get_reg_name` functions to use lookup tables and allow to return .new names.

Fix build errors.

Move lookup table names to PluginInfo; Fix off by one error.

Add helper function to generate doxygen

Add ALIAS2REG function.

Set to latest HEAD.

Generate flags correctly

Update rzil-compiler

Update generated files.

Remove left duplex code from rebase.

Add helper functions for IL op shuffle functionality.

Add "dont compile option."

Add shuffle function for IL ops.

This function brings the IL ops in the correct liniar order.

Update rzil-compiler

Log not compiled instructions.

Temporal remove compiler submodule.

Clone submodule rzil-compiler again

Filter pseudo instructions before compiling to show real insn count.

Update rzil-compiler

Update rzil-compiler

Update generated files.

Generate lookup table for IL getter in analysis plugin not asm plugin

Update docs

Move register names lookup tables into own header file and include it only in hexagon.c

Update generated files.

Update rzil-compiler

Update generated il ops files.

Set hardware loop in packet.

Add getter for il conifg

Introduce get_il_op

Add docs

Formatting, set alias C9 to PC, exclude op_builder

Update generated files.

Fix bug which added LLVM header to each IL file if IL files were skipped during generation.

* The skippped IL files were not added to the `unchanged_files` list and were not skipped during LLVM header adding.

Clearify what --no-rzil-compiler means.

Add endloop RZIL ops

Fix il_config getter names.

Replace const with RZ_BORROW

Several cleanups, renaming etfc.

Add HEX_REGFIELD macro.

Update rzil-compiler

Fix function generation: Remove code duplicates and fix getter names.

Update rzil compiler

Use only chars to identifier the ISA register and immediates no strings.

Formatting

Fix variable names

Fix include of NOT_IMPLEMTENTED. It was undefed before it was used.

Move NOT_IMPLEMENTED macro into rz-il-op-builder

Don't try to include deleted file.

Add uplifted extract64 and sextract64.

Include `hexagon_arch.h` for ISA to real imm/reg conversion functions

Don't define hexagon instruction if not in use.

Define Macros for decrement and increment

Update rzil-compiler

Add DEPOSIT64

update rzil-compiler

Move hex_get_rf_property_val to IL code

Fix typo from Il to IL

Allow to return _tmp from alias lookup tables

Implement get_npc

Let IL qemu functions return effects.

Print stats how many HVX and normal instructions were compiled.

Fix fall through cases

Add missing argument to macros

Update hand written RZIL funcitons

Don't init hi if it isn't used by instruciton.

Check for hi variable before initializing it.

Fix typos

Fix endloops after empty sequences are allowed.

Create own file for static il getter table

Fix typos

Add else case to leave r not uninitialized

Fix visibility and read only properties

Fix include

Add clo and clz

Increase visibility of get_pkt

hex_write_pred

Update rz_hexagon_il_config

Add sync register getter

CHeck for invalid instruction. Not valid instructions

Always execute packet which is at VM init address.

Update rzil-compiler

Sort insn id enums and il getter lookup tables so they are aligned.

Fix build errors of redeined variable.

Make get_hic globally visible

Fix up DUplex il ops getter.

Replace rz-list with rz_vector.

Fix ff by one

 Replace old list code

Add NULL checks

Set isa_id frmo template

Pass reg class not type: Set correct size when init rz-vector; FOrmatting

Copy bug fixes from Rizin.

Update endloop instructions.

Generate register profile without overlapping registers.

Decrease address size to 32bits for now.

Always use PC instead of C9

Update endloop functions

Add NULL check

Implement proper il op switch. FIxes invalid read.

Check for operand type during ISA -> OP conversion.

Fix double free

Set unsigned of valid length.

Update enloops

Mark packet after jump as valid.

Fix invlaid value types

Update helper functions.

Print compiled instructions ni progress bar.

Fix typo

Update compiler

Remove rebase duplication.

Correct grammar

Update non instruction IL code

Add predicate written flags.

Update compiler

Add functions to write registers and update double regs.

Fix register write.

Reset Pred write flags after executed instructions

Fix C4 write

Ensure that tmp prediacte register is written.

Update compiler

Update generated files.

Update access time of packet on request.

Add warnings i IL op failed.

Check for jumps before buffered packet is returned.

Check for _tmp predicate register names.

Remove PRED_WRITE function

Update compiler

Remove empty file

Remove specific clang-format version

Use main branch for rzil compiler..

Update RzIL-Compiler

Imported and LLVM encodings if both are present.

Run black

Update compiler

Update generated files

Rename p -> pkt and formatting

Set slot in HexInsn.

Handle slot_cancelled during runtime

Rename rzil_compiler package

Update rzil_compiler

Fix type name

Replace functoin call with macro

Update rzil_compiler

Fix macros

Update compiler

Update setuptools

Remove debug printf

Update handwritten code from Rizin commits

Parse shortcode before instruction parsing.

Move constants to typedefs.h

Add macros and function declarations for operand read/write

Add TransformedInstruction.

Moves a lot of IL instruction logic to the compiler.

Update hand-written files.

Add missing C20-C29 registers

Access rzil getter correctly

Shorten log message.

Add missing semicolion.

Remove wrong alternative names for C20-C29

Add alias_to_op and explicit_to_op implementations.

Add implementation for IL regs read/writes

Add macros.h again to hexagon.h and move the previously defined macros to this file.

Implement canceling slots

Remove const from pkt initialization.

Implement compilation of instructions only defined used in QEMU and arbitray code.

Add known sub_routines to the compiler.

Add more sub-routines and add support for circ_add.

Set RxV in fcircadd

Pass the bundle to fcircadd

Fix tests

Add const qualifier remove bracket

Remove unused functions.

Change visibility to API since those functions need to be reachable from analysis.

Remove px_written pseudo registers. They are tracked in the packet.

Use tracked read/writes to check if reg needs a sync.

Update compiler

Finish implementation of register read/writes.

Rename to commit_packet and il_ops_stats

Log reads/writes from low registers as well.

Fix assert reached: Cast values before LOGAND

Move sub-routine defintions to compiler.

Use jump sub-rutines

Set C9 if no jump did it.

Don't buffer invalid 0x00000000 instructions due to IO intransparency.

append jumpt to packet end, if there is no direct jump instruction in the packet.

Use macro value

Check for invalid instructions to determine packet emu readiness.

Print correct error.

Simplify packet commit to not add additional EMPTY() every time.

Add a jump_flag to determine if the jump to the next packet should happen.

Only init op if needed.

Fix setup of final effect sequence.

The setup-order is now 0...n, n...0 which broke the register write tracing.

Replace output filepaths with Path objects and ignore them in git.

Add getter for N-regs

Fix flag check

Add syntax of instruction as comment for searching.

Add count leading zeros/ones as sub-routines.

Print missing functions

Remove unnecessary parameters from log read/write reg functions

Use correct shift amount for doub reg writes.

Remove unnecessary AND

Handle case when registers are read and written.

The returned value in this case must be the .new value after the first write.

Update compiler

Resolve register numbers from asm version to enum id before using them

Set correct flags.

Update rzil_compiler

Add operators like &= to syntax coloring

Track predicate writes for each slot.

Allow read of C9 (PC) which is not present in the VM.

Update compiler

Increase number of state packets since 8 are to little and break rz-tracetest.

Remove incorrect increment

Update compiler

Hopefully fix the annoying buffering poblem for the last time.

Parse assignments like &= in asm string

Return NULL or U32 from get_reg_field

Update compiler

Add debug printing function to trace acces to buffer of packets.

Use rizin mem read function to get int from buffer

Add more detail to debug state printing

Fix add to packet algorithm

The decision to what packet the instruction
should be added was too early. Instructions
which belonged to another packet were added to stale one.

Remove PS_ instructions.

Remove unused function.

Handle MOD registers in il reg read/write

Handle writes to immutable and partly immutable registers.

Handle read/write of alias P3:0 register

Add resolve function for Mod to CS regs

Add struct members for floats.

Add macros for floats

Update compiler

Run black

Update license info

Several tiny syntactical and style fixes

Reduce indentation by inverting NULL check.

Add missing newlines before function defintions.

Replace string path with Path().

Fix rw overlap check by only performing it on x register.

Add RzIL tests.

Update generated files

Add missing asm tests

Return NULL if register name could not be resolved and mark isntructoin as invalid.

Decrease log level as it spamms during aaa

Update analysis tests.

Update CC generation with upper case reg names

Fix UndefinedBehaviorSanitizer error for int is shifted by 31

Fix resource leaks

Reduce log level for invalid duplex classes.

Those get often hit by disassembling invalid instructions.

Only generate IL code if requested.

Enable load_align test.

Fix syntax: Remove trailing ; for invalid decodes with parse_bits == 0.

Add angle brakcets type annotations.

Add NULL check for asm_toks

Add updated pcre2 regex patterns

Sync from rizin branch

Sync with C source.

Fix includes with new RzArch refactoring

Remove useless comments
  • Loading branch information
Rot127 committed Mar 15, 2024
1 parent 93849a7 commit afb1685
Show file tree
Hide file tree
Showing 66 changed files with 4,113 additions and 698 deletions.
6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ rz_hexagon.egg-info/
Hexagon.json
.config
.last_llvm_commit_info
venv/
.venv
/rizin/
.venv/
.vscode
rizin/
4 changes: 4 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[submodule "rzil_compiler"]
path = rzil_compiler
url = [email protected]:Rot127/rzil-compiler.git
branch = main
8 changes: 8 additions & 0 deletions .reuse/dep5
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ Files: .gitignore
Copyright: 2021 RizinOrg <[email protected]>
License: LGPL-3.0-only

Files: .gitmodules
Copyright: 2022 Rot127 <[email protected]>
License: LGPL-3.0-only

Files: .pylintrc
Copyright: 2021 RizinOrg <[email protected]>
License: LGPL-3.0-only
Expand Down Expand Up @@ -54,3 +58,7 @@ License: LGPL-3.0-only
Files: import/*
Copyright: 2022 Rot127 <[email protected]>
License: LGPL-3.0-only

Files: handwritten/*.json
Copyright: 2023 Rot127 <[email protected]>
License: LGPL-3.0-only
75 changes: 75 additions & 0 deletions Conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# SPDX-FileCopyrightText: 2022 Rot127 <[email protected]>
# SPDX-License-Identifier: LGPL-3.0-only

import subprocess

from enum import StrEnum
from pathlib import Path

from helperFunctions import log


class OutputFile(StrEnum):
"""
Enum of paths used by the components.
<REPO> is replaced with the path to the repositories root.
<ARCH> is replaced with the architecture name.
"""

OUT_BASE = "<REPO>/rizin/"
LIBRZ_DIR = "<REPO>/rizin/librz/"
IL_OPS_DIR = "<REPO>/rizin/librz/analysis/arch/hexagon/il_ops/"

ANA_TESTS = "<REPO>/rizin/test/db/analysis/hexagon"
ASM_TESTS = "<REPO>/rizin/test/db/asm/hexagon"
RZIL_TESTS = "<REPO>/rizin/test/db/rzil/hexagon"
ANALYSIS_HEXAGON_C = "<REPO>/rizin/librz/analysis/p/analysis_hexagon.c"
ASM_HEXAGON_C = "<REPO>/rizin/librz/asm/p/asm_hexagon.c"
CC_HEXAGON_32_SDB_TXT = "<REPO>/rizin/librz/analysis/d/cc-hexagon-32.sdb.txt"
HEXAGON_IL_C = "<REPO>/rizin/librz/analysis/arch/hexagon/hexagon_il.c"
HEXAGON_IL_GETTER_TABLE_H = "<REPO>/rizin/librz/analysis/arch/hexagon/hexagon_il_getter_table.h"
HEXAGON_IL_H = "<REPO>/rizin/librz/analysis/arch/hexagon/hexagon_il.h"
HEXAGON_ARCH_C = "<REPO>/rizin/librz/asm/arch/hexagon/hexagon_arch.c"
HEXAGON_ARCH_H = "<REPO>/rizin/librz/asm/arch/hexagon/hexagon_arch.h"
HEXAGON_C = "<REPO>/rizin/librz/asm/arch/hexagon/hexagon.c"
HEXAGON_DISAS_C = "<REPO>/rizin/librz/asm/arch/hexagon/hexagon_disas.c"
HEXAGON_H = "<REPO>/rizin/librz/asm/arch/hexagon/hexagon.h"
HEXAGON_INSN_H = "<REPO>/rizin/librz/asm/arch/hexagon/hexagon_insn.h"
HEXAGON_REG_TABLES_H = "<REPO>/rizin/librz/asm/arch/hexagon/hexagon_reg_tables.h"
HEXAGON_DWARF_REG_TABLE_H = "<REPO>/rizin/librz/analysis/hexagon_dwarf_reg_num_table.inc"


class Conf:
"""
Holds all the configurable values like paths.
"""

@staticmethod
def replace_placeholders(path_str: str) -> str:
if "<REPO>" in path_str:
root = subprocess.run(
["git", "rev-parse", "--show-toplevel"],
check=True,
stdout=subprocess.PIPE,
)
root_dir = Path(root.stdout.decode("utf8").strip("\n"))
if not root_dir.exists():
raise NotADirectoryError(str(root_dir))

path_str = path_str.replace("<REPO>", str(root_dir))
return path_str

@staticmethod
def get_path(file: OutputFile) -> Path:
return Path(Conf.replace_placeholders(file))

@staticmethod
def check_path(path: Path, is_file: bool = True) -> None:
"""Checks a given path and creates the directory if it doesn't exist."""
if not path.exists():
target = path
if is_file:
target = path.parent
log(f"Create dir {str(target)}")
target.mkdir(parents=True, exist_ok=True)
4 changes: 4 additions & 0 deletions HardwareRegister.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,10 @@ def set_well_defined_asm_names(self, llvm_asm: str, llvm_alt: list):
match_alias = re.search(r"^[rcpgvqs]\d{1,2}(:\d{1,2})?$", ",".join(llvm_alt))
if (llvm_asm == "p3:0") or (llvm_asm in llvm_alt):
match_asm = None
if (llvm_asm in llvm_alt) and len(llvm_alt) == 1:
# Alias of some regs equal asm name.
self.asm_name = llvm_asm
self.alias = []
if match_asm and match_alias:
raise ImplementationException(
"HW reg alias and asm names match same pattern: alias: {} asm: {}".format(",".join(llvm_alt), llvm_asm)
Expand Down
18 changes: 13 additions & 5 deletions Immediate.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,15 @@ def __init__(
self.encoding_width = 0 # Num. bits stored in encoding.
self.total_width = 0

self.parse_imm_type(llvm_type)

def parse_imm_type(self, llvm_imm_type: str) -> None:
"""Parse immediate types like: u4_2Imm. This method sets all kinds of flags, the scale and total width."""
self.parse_imm_type(llvm_type, llvm_syntax == "II")

def parse_imm_type(self, llvm_imm_type: str, is_second: bool) -> None:
"""Parse immediate types like: u4_2Imm. This method sets
all kinds of flags, the scale, total width and ISA identifier.
Args:
llvm_imm_type: The llvm type string (e.g.: u4_2Imm).
is_second: Flag if this immediate is the second immediate in the instruction.
"""
type_letter = re.search(r"^([a-z]+)\d{1,2}", llvm_imm_type)
if not type_letter:
raise ImplementationException("Unhandled immediate type: {}".format(llvm_imm_type))
Expand All @@ -80,6 +85,7 @@ def parse_imm_type(self, llvm_imm_type: str) -> None:
elif type_letter == "a" or type_letter == "b":
self.is_signed = True
self.is_pc_relative = True
type_letter = "r" # In QEMUs shortcode all PC relative immediates are named with 'r'
# Constant value -1
elif type_letter == "n":
self.is_signed = True
Expand All @@ -95,6 +101,7 @@ def parse_imm_type(self, llvm_imm_type: str) -> None:
return
else:
raise ImplementationException("Unhandled immediate type: {}".format(llvm_imm_type))
self.isa_id = type_letter.upper() if is_second else type_letter

# Value before _ represents number of encoded bits.
result = re.search(r"[a-z](\d+)\_", llvm_imm_type)
Expand Down Expand Up @@ -147,7 +154,8 @@ def c_template(self, force_extendable=False) -> str:
if self.total_width == 32:
info.append("HEX_OP_TEMPLATE_FLAG_IMM_DOUBLE_HASH")
info = " | ".join(info)
r = f".info = {info}, .masks = {{ {self.opcode_mask.c_template} }}"
r = f".info = {info}, .masks = {{ {self.opcode_mask.c_template} }}, "
r += f".isa_id = '{self.isa_id if self.isa_id != '' else 0}'"
if self.scale > 0:
r += f", .imm_scale = {self.scale}"
return r
9 changes: 7 additions & 2 deletions InstructionTemplate.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import HexagonArchInfo
import PluginInfo
from rzil_compiler.Compiler import RZILInstruction
from Immediate import Immediate
from ImplementationException import ImplementationException
from InstructionEncoding import InstructionEncoding
Expand Down Expand Up @@ -87,7 +88,7 @@ def __init__(self, llvm_instruction):
self.llvm_new_operand_index: bool = None
self.is_predicated: bool = False
self.is_pred_new: bool = False
self.is_pred_false: bool = False # Duplex can have both, true and false predicates.
self.is_pred_false: bool = False
self.is_pred_true: bool = False

# Special
Expand All @@ -97,6 +98,8 @@ def __init__(self, llvm_instruction):
self.is_loop_begin: bool = None
self.loop_member = None

self.il_ops: RZILInstruction = None

# Execution specific (Interesting for decompiler plugin)
# The address mode of load/store instructions
self.addr_mode = None
Expand Down Expand Up @@ -216,6 +219,8 @@ def parse_instruction(self) -> None:
# Indices of new values (stored in "opNewValue") are only for non predicates.
is_new_value = self.new_operand_index == index and self.has_new_non_predicate
operand = Register(op_name, op_type, is_new_value, index)
# Second letter in reg name is used in QEMU shortcode to identify the register.
operand.isa_id = op_name[1]
# Whether the predicate registers holds a new value is denoted in "isPredicatedNew".
if self.is_pred_new and operand.is_predicate:
operand.is_new_value = True
Expand Down Expand Up @@ -314,7 +319,7 @@ def get_template_in_c(self) -> str:
flags.append("HEX_INSN_TEMPLATE_FLAG_LOOP_0")
elif self.loop_member == LoopMembership.HEX_LOOP_1:
flags.append("HEX_INSN_TEMPLATE_FLAG_LOOP_1")
if flags != []:
if flags:
flags = " | ".join(flags)
code += f".flags = {flags},\n"
code += "}"
Expand Down
Loading

0 comments on commit afb1685

Please sign in to comment.