Skip to content

Commit

Permalink
Use libnuma instead of hwloc
Browse files Browse the repository at this point in the history
It's way, way simpler.
  • Loading branch information
joelsmithTT committed Nov 26, 2024
1 parent 004112c commit 7231812
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 59 deletions.
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Usermode Driver for Tenstorrent AI Accelerators
## Dependencies
Required Ubuntu dependencies:
```
sudo apt install -y libhwloc-dev cmake ninja-build
sudo apt install -y cmake ninja-build libnuma-dev
```

Suggested third-party dependency is Clang 17:
Expand All @@ -17,7 +17,7 @@ sudo ./llvm.sh 17

## Build flow

To build `libdevice.so`:
To build `libdevice.so`:
```
cmake -B build -G Ninja
cmake --build build
Expand Down Expand Up @@ -70,7 +70,7 @@ add_subdirectory(<path to umd>)

### Ubuntu
```
apt install ./umd-dev-x.y.z-Linux.deb
apt install ./umd-dev-x.y.z-Linux.deb
```

## Simulator Integration
Expand Down Expand Up @@ -100,7 +100,7 @@ To set up pre-commit on your local machine, follow these steps:
Ensure you have Python installed, then run:
```bash
pip install pre-commit
```
```
2. **Install the Git Hook Scripts**:
In your local repository, run the following command to install the pre-commit hooks:
```bash
Expand All @@ -113,10 +113,10 @@ To set up pre-commit on your local machine, follow these steps:
pre-commit run --all-files
```
## Why You Should Use Pre-commit
By setting up pre-commit locally, you can help maintain the quality of the codebase and ensure that commits consistently meet the project's formatting standards. This saves time during code reviews and reduces the likelihood of code formatting issues slipping into the repository.
Since the hooks run automatically before each commit, you don't need to remember to manually format or check your code, making it easier to maintain consistency.
By setting up pre-commit locally, you can help maintain the quality of the codebase and ensure that commits consistently meet the project's formatting standards. This saves time during code reviews and reduces the likelihood of code formatting issues slipping into the repository.

Since the hooks run automatically before each commit, you don't need to remember to manually format or check your code, making it easier to maintain consistency.

We strongly encourage all developers to integrate pre-commit into their workflow.

# Formatting C++ code
Expand Down
2 changes: 1 addition & 1 deletion device/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ target_link_libraries(
PUBLIC
flatbuffers
PRIVATE
hwloc
numa
nng
rt
uv_a
Expand Down
58 changes: 8 additions & 50 deletions device/pcie/pci_device.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@

#include "pci_device.hpp"

#include <fcntl.h> // for ::open
#include <hwloc.h>
#include <fcntl.h> // for ::open
#include <linux/pci.h> // for PCI_SLOT, PCI_FUNC
#include <numa.h>
#include <sys/ioctl.h> // for ioctl
#include <sys/mman.h> // for mmap, munmap
#include <sys/stat.h> // for fstat
Expand Down Expand Up @@ -46,53 +46,6 @@ const uint32_t HUGEPAGE_REGION_SIZE = 1 << 30; // 1GB
using namespace tt;
using namespace tt::umd;

/**
* Configures the NUMA memory binding for the current process.
*
* Sets the memory allocation policy to bind memory allocations to the specified
* NUMA node for the lifetime of the process.
*
* @param numa_node The NUMA node to which memory allocations should be bound
*/
static void configure_numa_binding(int numa_node) {
if (numa_node < 0) {
// Don't bother if we're not on a NUMA system.
return;
}

hwloc_topology_t topology;

if (hwloc_topology_init(&topology) < 0) {
log_warning(LogSiliconDriver, "hwloc_topology_init failed; will not bind to NUMA node.");
return;
}

if (hwloc_topology_load(topology) < 0) {
log_warning(LogSiliconDriver, "hwloc_topology_load failed; will not bind to NUMA node.");
hwloc_topology_destroy(topology);
return;
}

hwloc_nodeset_t nodeset = hwloc_bitmap_alloc();
if (!nodeset) {
log_warning(LogSiliconDriver, "hwloc_bitmap_alloc failed; will not bind to NUMA node.");
hwloc_topology_destroy(topology);
return;
}

hwloc_bitmap_only(nodeset, numa_node);
if (hwloc_set_membind(topology, nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_THREAD) < 0) {
log_warning(LogSiliconDriver, "hwloc_set_membind failed: {}", strerror(errno));
log_warning(LogSiliconDriver, "Your hugepage memory may not be on the same NUMA node as the device.");
hwloc_bitmap_free(nodeset);
hwloc_topology_destroy(topology);
return;
}

hwloc_bitmap_free(nodeset);
hwloc_topology_destroy(topology);
}

template <typename T>
static T read_sysfs(const PciDeviceInfo &device_info, const std::string &attribute_name) {
const auto sysfs_path = fmt::format(
Expand Down Expand Up @@ -735,7 +688,12 @@ tt::umd::architecture_implementation *PCIDevice::get_architecture_implementation
bool PCIDevice::init_hugepage(uint32_t num_host_mem_channels) {
const size_t hugepage_size = HUGEPAGE_REGION_SIZE;

configure_numa_binding(this->numa_node);
if (numa_node > numa_max_node()) {
log_warning(LogSiliconDriver, "numa_node: {} is greater than numa_max_node: {}.", numa_node, numa_max_node());
}

// Prefer allocations on the NUMA node associated with the device.
numa_set_preferred(numa_node);

// Convert from logical (device_id in netlist) to physical device_id (in case of virtualization)
auto physical_device_id = get_device_num();
Expand Down

0 comments on commit 7231812

Please sign in to comment.