Skip to content

README.md

Latest commit

 

History

History
220 lines (160 loc) · 9.65 KB

README.md

File metadata and controls

220 lines (160 loc) · 9.65 KB
Raw

ex-opt 🐦

CI License: MIT C++ Language Standard

ex-opt is a high-performance, asynchronous, and distributed black-box optimization framework written in modern C++20. Built on top of the stdexec execution model and the ex-actor distributed actor framework, it enables seamless parallel evaluation of computationally expensive candidate states (e.g., quant strategy backtesting, physical simulations, ML hyperparameter tuning) across local CPU cores or a cluster of machines.

Key Features

  • Modern C++20 Architecture: Extensively utilizes coroutines, concepts, ranges, and span to deliver compile-time safety and expressive code.
  • Structured Concurrency (stdexec): Built on the C++ standard executor model (Sender/Receiver), providing lock-free, ultra-low overhead task scheduling and thread pool execution.
  • Unified Synchronous Interface (ex_opt::optimize): Easily execute optimization locally or distributed using a single, synchronous functional interface. No need to manually write async task orchestrators.
  • Distributed Actors (ex-actor): Evaluation tasks are encapsulated as actors. Local multi-threading and remote multi-node clustering share the exact same code and programming interface.
  • Simple Distributed Export (EX_OPT_REGISTER_DISTRIBUTED): A zero-boilerplate macro to automatically export and register user objective functions for ZeroMQ-based RPC across the cluster.
  • Two Concurrent Scheduling Pipelines:
    • Batch Pipeline: For population-based algorithms (e.g., Genetic Algorithms, Particle Swarm Optimization) where an entire generation of candidate parameters is evaluated in parallel.
    • Stream Pipeline: For sequential optimization algorithms (e.g., Simulated Annealing) utilizing a "refill-on-completion" pipeline. It dispatches new candidates as soon as any worker becomes free, preventing worker starvation due to "stragglers."
  • Decoupled Design: Complete separation of the optimization algorithm (Policy) and the simulation execution (WorkActor).

Supported Algorithms

ex-opt provides several built-in global optimization algorithms:

  1. Particle Swarm Optimization (PSO) (Batch mode) - particle_swarm_optimization.h
  2. Genetic Algorithm (GA) (Batch mode) - genetic_algorithm.h
  3. Simulated Annealing (SA) (Stream mode) - simulated_annealing.h

Quick Start (Local Multi-Threading)

Here is a simple example showing how to optimize a 5D Sphere function locally using Particle Swarm Optimization (PSO) and the unified optimize API.

#include <iostream>
#include <vector>
#include "ex_opt.h"
#include "policy/particle_swarm_optimization.h"

// 1. Define the objective function (lower is better)
static double SphereFunction(const std::vector<double>& x) {
  double sum = 0.0;
  for (double xi : x) sum += xi * xi;
  return sum;
}

int main() {
  // 2. Configure PSO Settings for 5D space bounded in [-10.0, 10.0]
  ex_opt::pso::PsoSettings settings(/*dim=*/5, /*range_lo=*/-10.0, /*range_hi=*/10.0);
  settings.goal = 1e-6;
  settings.steps = 1000;
  settings.print_every = 100;

  ex_opt::pso::ParticleSwarmPolicy<std::vector<double>> policy(std::move(settings));

  // 3. Run the optimization locally using 4 concurrent worker actors
  auto best = ex_opt::optimize(policy, SphereFunction, /*num_actors=*/4);

  std::cout << "Best parameters found: [";
  for (double val : best) std::cout << val << " ";
  std::cout << "]\nFinal Fitness: " << policy.GetSolver().GetResult().error << std::endl;

  return 0;
}

Distributed Optimization (Clustering)

ex-opt enables seamless transition from local multi-core optimization to a cluster of machines.

1. Register Objective Function

To make your objective function callable across remote worker processes, register it in your executable's global scope using the helper macro:

#include "ex_opt.h"

static double MyObjective(const std::vector<double>& x) { ... }

// Registers the objective function internally for distributed ZeroMQ RPC
EX_OPT_REGISTER_DISTRIBUTED(MyObjective, std::vector<double>);

2. Startup Worker Nodes

On your worker machines, run the application in worker role. Command-line arguments can be parsed using ex_opt::ParseArgs:

./my_app --role worker --listen tcp://127.0.0.1:5555 --join tcp://127.0.0.1:5556 --name worker1

3. Startup Manager Node

On your main manager machine, run the application in manager role and specify worker node addresses:

./my_app --role manager --listen tcp://127.0.0.1:5556 --join tcp://127.0.0.1:5555 --name manager --workers tcp://127.0.0.1:5555 --actors 2

Alternatively, load configuration settings into ExOptConfig programmatically:

ex_opt::ExOptConfig config;
config.role = ex_opt::ExOptConfig::Role::Manager;
config.listen_address = "tcp://127.0.0.1:5556";
config.workers = {"tcp://127.0.0.1:5555"};
config.local_num_actors = 2;

auto best = ex_opt::optimize(policy, MyObjective, std::move(config));

Performance & Scalability Benchmark

We evaluated ex-opt's scaling efficiency on a real-world bare-metal cluster (8 nodes, Intel Xeon Gold 6226R @ 2.90GHz, 16 physical cores / 32 threads per node, 10GbE network interconnect).

  • Objective Function: 10-Dimensional Rastrigin Benchmark Function
  • Swarm Size: 4096 particles per generation
  • Iterations: 20 steps (generations)
  • Workload Intensity: 100,000 iterations/eval (FPU/ALU compute-heavy)

Benchmark Scaling Performance

1. Single-Node Multi-Core Scaling (node1, 1 to 32 Threads)

Across physical cores (1 to 16), ex-opt maintains near-perfect linear scaling thanks to stdexec's lock-free task stealing and actor isolation:

Concurrency (Actors) Execution Time Throughput Speedup Parallel Efficiency Architecture Note
1 2942.6 s (~49 min) 27.84 evals/s 1.00x 100.0% Single-core baseline
2 1475.6 s 55.51 evals/s 1.99x 99.5% Near-perfect linearity
4 744.6 s 110.02 evals/s 3.95x 98.8% Linear scaling
8 375.3 s 218.30 evals/s 7.84x 98.0% Single NUMA socket
16 191.6 s 427.48 evals/s 15.36x 96.0% All 16 physical cores
32 130.2 s 629.07 evals/s 22.60x 70.6% SMT (Hyperthreading) +47.2% net throughput

2. Distributed Multi-Node Cluster Scaling (1 to 8 Nodes, 256 Total Actors)

Scaling horizontally across physical servers with 32 worker actors per node:

Cluster Setup Total Actors Elapsed Time Throughput Cluster Speedup Cluster Efficiency
1 Node (node1) 32 130.2 s 629.07 evals/s 1.00x 100.0%
2 Nodes 64 78.6 s 1042.33 evals/s 1.66x 82.8%
4 Nodes 128 45.1 s 1815.44 evals/s 2.89x 72.1%
8 Nodes 256 28.4 s 2887.96 evals/s 4.59x 57.4%

Overall Acceleration: Extending from 1 single core (2942.6s) to an 8-node cluster (28.4s) achieves an overall speedup of 103.7x, compressing an evaluation job that took nearly an hour down to under 29 seconds.

How to Build

Prerequisites

  • A modern C++20 compiler (GCC >= 11, Clang >= 13, MSVC >= 19.30)
  • CMake >= 3.25
  • Git (for fetching dependencies)

Cloning the Repository

git clone https://github.com/SYSUljz/ex-opt.git
cd ex-opt

Building the Project

Configure and compile the library, tests, and examples:

cmake -B build -S . -DEX_OPT_BUILD_EXAMPLES=ON -DEX_OPT_BUILD_TESTS=ON
cmake --build build -j$(nproc)

Running Tests & Examples

Run the GoogleTest suite:

cd build
ctest --output-on-failure

Run the PSO Sphere optimization example:

./examples/pso_sphere

Integrating into your Project

You can install ex-opt and import it into your own CMake project.

Installing ex-opt

cmake --install build --prefix /path/to/install

CMake Integration

Once installed, link it inside your CMakeLists.txt:

find_package(ex-opt CONFIG REQUIRED)

add_executable(my_optimization_app main.cc)
target_link_libraries(my_optimization_app PRIVATE ex_opt::ex-opt)

Future Roadmap (TODO)

Here are the planned features and improvements for ex-opt:

  • Python Bindings (pybind11): Allow defining objective functions in Python while running search algorithms and parallel orchestration in high-performance C++.
  • Algorithm Extensions: Add support for Covariance Matrix Adaptation Evolution Strategy (CMA-ES), Differential Evolution (DE), and multi-objective optimization (NSGA-II).
  • Asynchronous Swarm Optimization: Implement Async Particle Swarm Optimization (APSO) and Async Genetic Algorithm (AGA) to eliminate generational barrier bottlenecks.
  • Distributed Fault Tolerance: Automatically detect worker node disconnection and re-schedule in-flight candidate evaluations to other active nodes.

License

This project is licensed under the MIT License - see the LICENSE file for details. Some underlying algorithms are ported and adapted from scikit-opt and Kentzoglanakis' PSO implementation.