ex-opt 🐦

ex-opt is a high-performance, asynchronous, and distributed black-box optimization framework written in modern C++20. Built on top of the stdexec execution model and the ex-actor distributed actor framework, it enables seamless parallel evaluation of computationally expensive candidate states (e.g., quant strategy backtesting, physical simulations, ML hyperparameter tuning) across local CPU cores or a cluster of machines.

---

Key Features

• Modern C++20 Architecture: Extensively utilizes coroutines, concepts, ranges, and span to deliver compile-time safety and expressive code.
• Structured Concurrency (stdexec): Built on the C++ standard executor model (Sender/Receiver), providing lock-free, ultra-low overhead task scheduling and thread pool execution.
• Unified Synchronous Interface (ex_opt::optimize): Easily execute optimization locally or distributed using a single, synchronous functional interface. No need to manually write async task orchestrators.
• Distributed Actors (ex-actor): Evaluation tasks are encapsulated as actors. Local multi-threading and remote multi-node clustering share the exact same code and programming interface.
• Simple Distributed Export (EX_OPT_REGISTER_DISTRIBUTED): A zero-boilerplate macro to automatically export and register user objective functions for ZeroMQ-based RPC across the cluster.
• Two Concurrent Scheduling Pipelines:
  • Batch Pipeline: For population-based algorithms (e.g., Genetic Algorithms, Particle Swarm Optimization) where an entire generation of candidate parameters is evaluated in parallel.
  • Stream Pipeline: For sequential optimization algorithms (e.g., Simulated Annealing) utilizing a "refill-on-completion" pipeline. It dispatches new candidates as soon as any worker becomes free, preventing worker starvation due to "stragglers."
• Decoupled Design: Complete separation of the optimization algorithm (Policy) and the simulation execution (WorkActor).

---

Supported Algorithms

ex-opt provides several built-in global optimization algorithms:

1. Particle Swarm Optimization (PSO) (Batch mode) - particle_swarm_optimization.h
2. Genetic Algorithm (GA) (Batch mode) - genetic_algorithm.h
3. Simulated Annealing (SA) (Stream mode) - simulated_annealing.h

---

Quick Start (Local Multi-Threading)

Here is a simple example showing how to optimize a 5D Sphere function locally using Particle Swarm Optimization (PSO) and the unified optimize API.

#include <iostream>
#include <vector>
#include "ex_opt.h"
#include "policy/particle_swarm_optimization.h"

// 1. Define the objective function (lower is better)
static double SphereFunction(const std::vector<double>& x) {
  double sum = 0.0;
  for (double xi : x) sum += xi * xi;
  return sum;
}

int main() {
  // 2. Configure PSO Settings for 5D space bounded in [-10.0, 10.0]
  ex_opt::pso::PsoSettings settings(/*dim=*/5, /*range_lo=*/-10.0, /*range_hi=*/10.0);
  settings.goal = 1e-6;
  settings.steps = 1000;
  settings.print_every = 100;

  ex_opt::pso::ParticleSwarmPolicy<std::vector<double>> policy(std::move(settings));

  // 3. Run the optimization locally using 4 concurrent worker actors
  auto best = ex_opt::optimize(policy, SphereFunction, /*num_actors=*/4);

  std::cout << "Best parameters found: [";
  for (double val : best) std::cout << val << " ";
  std::cout << "]\nFinal Fitness: " << policy.GetSolver().GetResult().error << std::endl;

  return 0;
}

---

Distributed Optimization (Clustering)

ex-opt enables seamless transition from local multi-core optimization to a cluster of machines.

1. Register Objective Function

To make your objective function callable across remote worker processes, register it in your executable's global scope using the helper macro:

#include "ex_opt.h"

static double MyObjective(const std::vector<double>& x) { ... }

// Registers the objective function internally for distributed ZeroMQ RPC
EX_OPT_REGISTER_DISTRIBUTED(MyObjective, std::vector<double>);

2. Startup Worker Nodes

On your worker machines, run the application in worker role. Command-line arguments can be parsed using ex_opt::ParseArgs:

./my_app --role worker --listen tcp://127.0.0.1:5555 --join tcp://127.0.0.1:5556 --name worker1

3. Startup Manager Node

On your main manager machine, run the application in manager role and specify worker node addresses:

./my_app --role manager --listen tcp://127.0.0.1:5556 --join tcp://127.0.0.1:5555 --name manager --workers tcp://127.0.0.1:5555 --actors 2

Alternatively, load configuration settings into ExOptConfig programmatically:

ex_opt::ExOptConfig config;
config.role = ex_opt::ExOptConfig::Role::Manager;
config.listen_address = "tcp://127.0.0.1:5556";
config.workers = {"tcp://127.0.0.1:5555"};
config.local_num_actors = 2;

auto best = ex_opt::optimize(policy, MyObjective, std::move(config));

---

Performance & Scalability Benchmark

We evaluated ex-opt's scaling efficiency on a real-world bare-metal cluster (8 nodes, Intel Xeon Gold 6226R @ 2.90GHz, 16 physical cores / 32 threads per node, 10GbE network interconnect).

• Objective Function: 10-Dimensional Rastrigin Benchmark Function
• Swarm Size: 4096 particles per generation
• Iterations: 20 steps (generations)
• Workload Intensity: 100,000 iterations/eval (FPU/ALU compute-heavy)

1. Single-Node Multi-Core Scaling (node1, 1 to 32 Threads)

Across physical cores (1 to 16), ex-opt maintains near-perfect linear scaling thanks to stdexec's lock-free task stealing and actor isolation:

Concurrency (Actors)	Execution Time	Throughput	Speedup	Parallel Efficiency	Architecture Note
1	2942.6 s (~49 min)	27.84 evals/s	1.00x	100.0%	Single-core baseline
2	1475.6 s	55.51 evals/s	1.99x	99.5%	Near-perfect linearity
4	744.6 s	110.02 evals/s	3.95x	98.8%	Linear scaling
8	375.3 s	218.30 evals/s	7.84x	98.0%	Single NUMA socket
16	191.6 s	427.48 evals/s	15.36x	96.0%	All 16 physical cores
32	130.2 s	629.07 evals/s	22.60x	70.6%	SMT (Hyperthreading) +47.2% net throughput

2. Distributed Multi-Node Cluster Scaling (1 to 8 Nodes, 256 Total Actors)

Scaling horizontally across physical servers with 32 worker actors per node:

Cluster Setup	Total Actors	Elapsed Time	Throughput	Cluster Speedup	Cluster Efficiency
1 Node (node1)	32	130.2 s	629.07 evals/s	1.00x	100.0%
2 Nodes	64	78.6 s	1042.33 evals/s	1.66x	82.8%
4 Nodes	128	45.1 s	1815.44 evals/s	2.89x	72.1%
8 Nodes	256	28.4 s	2887.96 evals/s	4.59x	57.4%

Overall Acceleration: Extending from 1 single core (2942.6s) to an 8-node cluster (28.4s) achieves an overall speedup of 103.7x, compressing an evaluation job that took nearly an hour down to under 29 seconds.

---

How to Build

Prerequisites

• A modern C++20 compiler (GCC >= 11, Clang >= 13, MSVC >= 19.30)
• CMake >= 3.25
• Git (for fetching dependencies)

Cloning the Repository

git clone https://github.com/SYSUljz/ex-opt.git
cd ex-opt

Building the Project

Configure and compile the library, tests, and examples:

cmake -B build -S . -DEX_OPT_BUILD_EXAMPLES=ON -DEX_OPT_BUILD_TESTS=ON
cmake --build build -j$(nproc)

Running Tests & Examples

Run the GoogleTest suite:

cd build
ctest --output-on-failure

Run the PSO Sphere optimization example:

./examples/pso_sphere

---

Integrating into your Project

You can install ex-opt and import it into your own CMake project.

Installing ex-opt

cmake --install build --prefix /path/to/install

CMake Integration

Once installed, link it inside your CMakeLists.txt:

find_package(ex-opt CONFIG REQUIRED)

add_executable(my_optimization_app main.cc)
target_link_libraries(my_optimization_app PRIVATE ex_opt::ex-opt)

---

Future Roadmap (TODO)

Here are the planned features and improvements for ex-opt:

• Python Bindings (pybind11): Allow defining objective functions in Python while running search algorithms and parallel orchestration in high-performance C++.
• Algorithm Extensions: Add support for Covariance Matrix Adaptation Evolution Strategy (CMA-ES), Differential Evolution (DE), and multi-objective optimization (NSGA-II).
• Asynchronous Swarm Optimization: Implement Async Particle Swarm Optimization (APSO) and Async Genetic Algorithm (AGA) to eliminate generational barrier bottlenecks.
• Distributed Fault Tolerance: Automatically detect worker node disconnection and re-schedule in-flight candidate evaluations to other active nodes.

---

License

This project is licensed under the MIT License - see the LICENSE file for details. Some underlying algorithms are ported and adapted from scikit-opt and Kentzoglanakis' PSO implementation.