Ruby and Rust developer specializing in backend performance - Paweł Urbanek

Rustikon Talk - Rust Performance Profiling using hotpath-rs

Sun, 17 May 2026 22:49:07 +0200

Recently I gave a talk at the Rustikon conference about hotpath-rs - a Rust profiler focused on time, memory, CPU, and async performance debugging. I cover the project’s origin story and how it works under the hood. I also show a live instrumentation demo featuring the Zed editor.

It was my first conference talk in 8+ years, so I hope you enjoy it!

Links and resources

Lessons Learned Building High-Performance Rust Profiler

Tue, 12 May 2026 18:30:40 +0200

The Rust performance book features over a dozen different profiling tools. So I’m not sure if the world needed a new Rust profiler. Still, I spent the last 6+ months building hotpath-rs. In this post, I’ll describe the design decisions behind the library and share a few performance challenges I encountered while working on it. We’ll go deep into the low-level details: cache-line contention, async futures instrumentation, and decoding raw CPU traces back into Rust symbols.

hotpath profiler 101

The next section is a brief overview of the library. Click here if you want to jump straight into the implementation details.

Over the last months, the hotpath profiler has grown to over 100k downloads on crates.io and is slowly gaining more adoption in the Rust ecosystem.

Before diving into implementation details, let’s quickly look at what hotpath does and why I built it. It’s an “all-in-one” Rust profiler / debugging toolkit designed to quickly identify performance bottlenecks.

The core idea is to combine multiple sources of data into reports that are quick and easy to mentally parse. You need only two macros, hotpath::main and hotpath::measure, to get started with instrumenting your codebase.

Let’s see it in action:

examples/overview.rs

#[hotpath::measure]
fn sync_work() {
    let mut result: u64 = 1;
    for i in 0..20000 {
        result = result.wrapping_mul(black_box(i as u64).wrapping_add(7));
        result ^= result >> 3;
    }
}

#[hotpath::measure]
fn sync_alloc() {
    for _ in 0..1000 {
        let buf: Vec<u8> = vec![1; 1024];
        std::hint::black_box(&buf);
    }
}

#[hotpath::measure]
async fn async_sleep() {
    tokio::time::sleep(Duration::from_millis(10)).await;
}

#[tokio::main]
#[hotpath::main]
async fn main() {
    for _ in 0..1000 {
        sync_work();
        sync_alloc();
        async_sleep().await;
    }
}

This example features 3 functions, showcasing different modes of execution present in any Rust program:

sync_work - synchronous function that executes a CPU-bound code
sync_alloc - synchronous function that allocates memory
async_sleep - function that sleeps asynchronously. A bit artificial, but it’s meant to simulate a slow async I/O. Waiting for an SQL query or an HTTP endpoint would yield similar performance results.

Looking at the example, can you tell what is the REAL bottleneck?

Let’s see a hotpath report output before we answer this question:

cargo run --example overview --features='hotpath,hotpath-alloc,hotpath-cpu'

[hotpath] 15.61s | timing, alloc, cpu

timing - Function execution time metrics.
+------------------------+-------+----------+----------+---------+---------+
| Function               | Calls | Avg      | P95      | Total   | % Total |
+------------------------+-------+----------+----------+---------+---------+
| cpu_basic::main        | 1     | 15.62 s  | 15.62 s  | 15.61 s | 100.00% |
+------------------------+-------+----------+----------+---------+---------+
| cpu_basic::async_sleep | 1000  | 11.77 ms | 12.10 ms | 11.77 s | 75.42%  |
+------------------------+-------+----------+----------+---------+---------+
| cpu_basic::sync_work   | 1000  | 3.14 ms  | 4.59 ms  | 3.14 s  | 20.11%  |
+------------------------+-------+----------+----------+---------+---------+
| cpu_basic::sync_alloc  | 1000  | 1.23 µs  | 2.62 µs  | 1.23 ms | 0.01%   |
+------------------------+-------+----------+----------+---------+---------+

alloc-bytes - Exclusive allocation bytes by each function.
Total: 1.1 MB
+------------------------+-------+----------+----------+-----------+---------+
| Function               | Calls | Avg      | P95      | Total     | % Total |
+------------------------+-------+----------+----------+-----------+---------+
| cpu_basic::sync_alloc  | 1000  | 1.0 KB   | 1.0 KB   | 1000.0 KB | 89.28%  |
+------------------------+-------+----------+----------+-----------+---------+
| cpu_basic::main        | 1     | 120.1 KB | 120.1 KB | 120.1 KB  | 10.72%  |
+------------------------+-------+----------+----------+-----------+---------+
| cpu_basic::async_sleep | 1000  | 0 B      | 0 B      | 0 B       | 0.00%   |
+------------------------+-------+----------+----------+-----------+---------+
| cpu_basic::sync_work   | 1000  | 0 B      | 0 B      | 0 B       | 0.00%   |
+------------------------+-------+----------+----------+-----------+---------+

cpu - CPU sampling attribution per function (exclusive). 
+------------------------+---------+---------+
| Function               | Samples | % Total |
+------------------------+---------+---------+
| cpu_basic::sync_work   | 1915914 | 56.13%  |
+------------------------+---------+---------+
| cpu_basic::async_sleep | 14056   | 0.41%   |
+------------------------+---------+---------+
| cpu_basic::sync_alloc  | 1581    | 0.05%   |
+------------------------+---------+---------+
samply load /tmp/hotpath/61089-1778083683167502000/hp.json.gz

hotpath timing, alloc, and CPU usage report.

Optionally, you can run samply load to see an interactive call tree or flamegraph generated with samply:

We can identify a different bottleneck depending on which report section we analyze:

async_sleep - takes ~75% of total program execution time, but less than 1% of CPU.
sync_alloc - took less than 1% of processing time but is responsible for ~90% of memory allocations.
sync_work - burned ~56% of total CPU cycles but took only ~20% of processing time

I came up with this example to exaggerate the core idea behind hotpath’s approach to profiling: different performance signals tell different stories.

This distinction is especially important for async-heavy Rust programs, where “slow” often means waiting on I/O, timers, channels, or the runtime scheduler rather than actually burning CPU. Traditional CPU profilers like samply or perf can show where CPU time is spent, but they cannot reliably capture time futures spend suspended while waiting to be polled by the runtime.

While working on Rust performance issues, I often found myself jumping between multiple specialized tools - samply for CPU usage, cargo-instruments for allocations, and tokio-console for async behavior… hotpath was built to bring those different signals together into a single report that’s fast to read and easy to reason about.

Naturally, this famous XKCD comic felt very relevant…

Source: xkcd.com/927

Live profiling TUI

Another unique feature of hotpath is live profiling and debugging capabilities. You can see it in action by running:

ssh demo.hotpath.rs

It executes an interactive TUI program that displays its own live performance metrics.

hotpath-rs TUI displaying its own instrumented future performance and lifetime metrics

You can think of it as “debugging on steroids”. hotpath::channel!, hotpath::stream! and hotpath::future! macros allow you to inspect data flowing in your system in real time. The required code changes are minimal because macros return exactly the same types and are 100% no-op unless the --features=hotpath is enabled.

In my current workflow, I don’t use hotpath TUI extensively; I prefer the static one-off reports and A/B benchmarks. But if you’re working on a long-running interactive program, you might find it helpful.

Where I do find the TUI dashboard useful is for optimizing the hotpath itself. I maintain an instrumented fork of the Zed editor, and test every new feature of hotpath against it. Given the scale and complexity of Zed, I treat it as the ultimate stress test, ensuring that lib should work with most other Rust programs.

You can check out the fork with instructions on how to run it yourself from this PR, or see instrumented Zed in action in the short demo:

Hopefully, I have your curiosity now. Read on if you’re interested in how the library works under the hood and the techniques I use to keep it performant.

Profiling the profiler

The core mechanism behind hotpath instrumentation is RAII (Resource Acquisition Is Initialization) guards. Here’s a simplified explanation of what hotpath::measure macro compiles to:

#[hotpath::measure]
fn sync_alloc() {
    for _ in 0..1000 {
        let buf: Vec<u8> = vec![1; 1024];
        std::hint::black_box(&buf);
    }
}

becomes:

fn sync_alloc() {
    let _guard = MeasurementGuard::new();
    for _ in 0..1000 {
        let buf: Vec<u8> = vec![1; 1024];
        std::hint::black_box(&buf);
    }
}

And here’s a simplified implementation of the MeasurementGuard struct:

pub struct MeasurementGuard {
    name: &'static str,
    start: Instant,
    sender: Sender<Measurement>,
}

impl Drop for MeasurementGuard {
    fn drop(&mut self) {
        let duration = self.start.elapsed();

        let measurement = Measurement {
            name: self.name,
            duration: duration
        };

        let _ = self.sender.send(measurement);
    }
}

MeasurementGuard uses std::time::Instant to measure time between its creation and drop, which is determined by how long an instrumented function executes. It’s the so-called “wall-clock timing” approach to profiling programs, which is significantly different from CPU sampling. The primary difference is in the results you’ll get when measuring async functions, which are notoriously hard to benchmark using only CPU profiling. Check out this docs section for an in-depth analysis of these two approaches.

Fixing the MPSC channel bottleneck

The MeasurementGuard’s Drop trait implementation is the hottest hot path in the project. It’s executed inline with the instrumented code, so it directly impacts the performance of measured programs.

In the initial library versions, the code resembled the above implementation, i.e., measurements were sent for background processing one by one. I’ve measured (using benchmark_noop.rs) the per-instrumentation impact to be ~500ns. Far from ideal if you want to instrument functions that execute on a microsecond scale.

I’ve used samply to inspect overhead, and it produced the following output:

samply call tree Thread::unpark bottleneck

As you can see ~22%, of all CPU traces landed in the Thread::unpark method. Apparently, even if messages are sent to a crossbeam MPSC channel every 1 microsecond, a receiver thread enters the parked state and has to be unparked to receive the next message. The solution was to send messages in batches, gathered per thread. It completely eliminated the Thread::unpark samples and reduced the avg. overhead per measurement to ~80ns.

I did follow up on this issue, and found out that there’s a mismatch in how crossbeam-channel handles receivers depending on whether they are used with a select! macro. A directly subscribed receiver uses a spin wait with backoff logic to rectify exactly this Thread::park issue. But select! macro bypasses this mechanism and causes the receiver thread to park even if data arrives in microsecond-scale intervals. I’ve submitted a PR trying to address this. Not sure if it’s the correct solution, but if you’re using crossbeam_channel::select! macro with high-frequency data, then, batching will likely measurably improve performance.

BTW, sending instrumentation data to a separate thread for processing is the most performance-critical area in the hotpath’s codebase. It directly impacts the performance of benchmarked programs and affects measurements. If you have an idea for a data-sharing mechanism with lower overhead than crossbeam channels, plz get in touch!

Alloc metrics CPU cache busting

Measuring memory allocations uses a system that is slightly more complicated than RAII guards with an internal timer. I’ve borrowed the initial implementation from the allocation-counter crate.

It uses a thread-local stack to attribute allocations and deallocations to each thread:

pub(crate) struct ThreadAllocStats {
    pub(crate) tid: AtomicU64,
    pub(crate) alloc_bytes: AtomicU64,
    pub(crate) dealloc_bytes: AtomicU64,
}

const MAX_THREADS: usize = 256;

static THREAD_ALLOC_STATS: [ThreadAllocStats; MAX_THREADS] = {
    const INIT: ThreadAllocStats = ThreadAllocStats::new();
    [INIT; MAX_THREADS]
};

A static preallocated array of AtomicU64, one per thread. When the hotpath-alloc feature is enabled, we hook into a system allocator and proxy allocations and deallocations through our custom CountingAllocator:

cfg_if::cfg_if! {
    if #[cfg(feature = "hotpath-alloc")] {
        #[global_allocator]
        static GLOBAL: functions::alloc::allocator::CountingAllocator = functions::alloc::allocator::CountingAllocator {};
    }

    unsafe impl GlobalAlloc for CountingAllocator {
        unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
            crate::lib_on::functions::alloc::core::track_alloc(layout.size());

            unsafe { System.alloc(layout) }
        }

        unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
            crate::lib_on::functions::alloc::core::track_dealloc(layout.size());

            unsafe {
                System.dealloc(ptr, layout);
            }
        }
    }
}

The GlobalAlloc trait defines alloc and dealloc, called for EVERY single allocation and deallocation in the instrumented Rust program.

That’s how we can attribute total alloc/dealloc stats for each thread. I use the benchmark_alloc.rs script to measure the overhead of an exaggerated function that does nothing but allocate memory:

#[hotpath::measure]
fn alloc() {
    for _ in 0..1000 {
        let vec = vec![1u8; 128];
        std::hint::black_box(vec);
    }
}

Here’s the benchmark execution time depending on the number of threads:

cargo build --example benchmark_alloc --features='hotpath,hotpath-alloc' --release 
hyperfine --warmup 3 'HOTPATH_ALLOC_NUM_THREADS=$THREADS_NUM ./target/release/examples/benchmark_alloc'

Threads	Time (ms)
1	~210
2	~720
4	~1000
8	~3700
12	~4400

This benchmark simulates an extreme case of a Rust program that does nothing but allocate. Surprisingly, the initial implementation worked without issues, even with the Zed debug build, which spawns ~60 threads. But you wouldn’t believe, a simple trick to instantly speed up Rust programs with multiple threads competing for a shared static resource:

#[repr(align(64))]
pub(crate) struct ThreadAllocStats {
    /// Thread ID (0 unused)
    pub(crate) tid: AtomicU64,
    pub(crate) alloc_bytes: AtomicU64,
    pub(crate) dealloc_bytes: AtomicU64,
}

Here are the same benchmark results after prepending ThreadAllocStats with #[repr(align(64))] annotation:

Threads	Before (ms)	After (ms)	% Improvement
1	~210	~210	0%
2	~720	~215	70%
4	~1000	~280	72%
8	~3700	~500	86%
12	~4400	~660	85%

This technique, called “cache padding”, forces each ThreadAllocStats instance to start at a 64-byte boundary, matching a common CPU cache-line size. Without padding, ThreadAllocStats in memory size was 24 bytes, so two instances could fit in a single CPU cache line. With multiple threads invalidating each other’s caches, performance was getting worse with each added thread. Check out the story of optimizing zlib-rb implementation for an in-depth exploration of this technique.

Async functions memory tracking

Another challenge was tracking memory allocations for instrumented functions. For synchronous functions, the implementation is still largely based on the allocation-counter crate. It uses thread_local data (AllocationInfoStack if you’re interested in the impl details) for tracking allocations. We use a RAII guard pattern similar to wall-clock time measurement:

impl AllocMeasurementGuard {
    pub fn new(name: &'static str, sender: Sender<Measurement>) -> Self {
        push_alloc_frame();
        Self { name, sender }
    }
}

impl Drop for AllocMeasurementGuard {
    fn drop(&mut self) {
        let (bytes, count) = pop_alloc_frame();
        let _ = self.sender.send(Measurement { 
          name: self.name, bytes, count 
        });
    }
}

It’s a simplified implementation to explain how it works. We maintain a thread_local stack and use push_alloc_frame to mark which synchronous function is currently executing. pop_alloc_frame marks functions as completed and reads allocation metrics for background processing.

But async Rust functions, depending on the runtime implementation, may migrate between threads. For example, the default #[tokio::main] macro spawns threads for each virtual CPU. You can easily check it in the Threads tab of a hotpath TUI:

hotpath TUI displays per-thread CPU and memory metrics

So, for async functions our thread-local data won’t work. Allocation info gathered for the function on thread A is no longer available if it executes pop_alloc_frame on a thread B.

The previous version of hotpath allowed async allocations tracking only for #[tokio::main(flavor = "current_thread")] runtime. I thought that as long as we use a single thread, measurements should be accurate. Unit tests returned correct results, and it supposedly worked. But benchmarking an Axum server app using the current_thread runtime, under a multithreaded oha loadtest, produced “interesting” results.

Can you spot an issue here?

A ~60s benchmark supposedly churned ~400GB of memory with a single routes::serve_doc_page method call allocating ~23MB on avg.

The bug was caused by using an RAII guard for instrumenting an asynchronous function, which, contrary to synchronous calls, does not execute continuously. Runtime can yield control of a future at any time and start executing a different one, even on a single thread. It caused the allocation metrics of measured functions to intertwine, massively inflating the numbers.

The solution was to instrument async functions as futures, which they really are. The current version of hotpath uses a similar implementation for tracking async memory usage:

pub struct InstrumentedFuture<F: Future> {
    #[pin]
    inner: F,
    sender: Option<&'static Sender<FutureEvent>>,
    future_id: u32,
    completed: bool,
}

impl<F: Future> Future for InstrumentedFuture<F> {
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
          crate::functions::alloc::guard::push_alloc_stack();
          this.inner.poll(cx)
          let (bytes, count) = crate::functions::alloc::guard::pop_alloc_stack();
          let _ = self.sender.send(Measurement { name: self.future_id, bytes, count });
    }
}

We wrap every instrumented function/future into an InstrumentedFuture struct, which holds a pinned reference (inner) to the original future. The hotpath::measure macro handles all the wrapping. And we push/pop the allocations data stack when runtime calls poll on our wrapper. Counterintuitively, the poll function is synchronous, so it’s guaranteed to execute continuously, and our allocation info no longer gets mixed up with other asynchronous calls.

And here’s the same benchmark after a fix:

This mechanism does not use any runtime-specific APIs, nor does it require setting additional config like RUSTFLAGS="--cfg tokio_unstable", or any changes to your project other than the hotpath::measure macro. The newest version of hotpath can correctly attribute memory allocations regardless of which runtime you’re using, tokio, smol, or anything custom.

CPU profiling integration

CPU profiling is a recent addition to the library. The core idea behind hotpath is combining different performance signals into easy-to-understand reports. Wall-clock timing and memory allocation metrics are useful on their own, but they don’t show the full picture.

An initial prototype used the pprof-rs crate, allowing sampling of CPU execution within the same process. I liked the simplicity of this approach, but it did not scale. The debug build of Zed slowed down to a crawl with pprof-rs profiling enabled.

A solution was to integrate samply profiler. It’s running in a separate process, so it works even with the debug profile Zed. Integration required a hacky workaround, because launching samply record --pid from the same process or its direct child caused it to get stuck. My best guess is that attaching the profiler temporarily suspends the target process and all its children. A workaround consists of a separate hotpath-samply binary, that spawns in a detached mode, and only then can we launch samply record without crashing the host process. “It’s broken, but it works”, and hotpath-cpu mode is available since the recent 0.16.0 release.

One caveat is that on Linux you have to prefix profiled programs with setsid -w, otherwise they exit before a final report is produced. See docs for more info.

Interpreting raw CPU traces

The host process and hotpath-samply communicate via a custom file-based API. Once hotpath allocations and timing profiling are completed, lib creates a stop-profiling file in a predefined location, and it communicates that we should also finish the CPU sampling by sending a SIGINT signal to samply. Later a done file creation communicates that the hp.json.gz file with profiling data is ready to be parsed. I came up with the hacky protocol by chatting with my AI friends. Let me know if you can think of a cleaner way to establish cross-process communication.

hp.json.gz is where things start to get interesting. This file contains raw data about CPU samples collected by samply profiler. Very raw:

{
  "threads": [{
    "name": "cpu_nested",
    "tid": 218490,
    "isMainThread": true,

    "stringArray": ["cpu_nested", "0x0ea0", "0x0c10"],

    "resourceTable": {
      "length": 1,
      "name": [0],
      "lib": [0],
      "type": [1]
    },

    "funcTable": {
      "length": 2,
      "name": [1, 2],
      "resource": [0, 0]
    },

    "frameTable": {
      "length": 2,
      "address": [0x0ea0, 0x0c10],
      "func": [0, 1],
      "nativeSymbol": [null, null]
    },

    "stackTable": {
      "length": 2,
      "prefix": [null, 0],
      "frame": [0, 1]
    },

    "samples": {
      "length": 1,
      "stack": [1],
      "timeDeltas": [3.12],
      "threadCPUDelta": [950],
      "weight": [1]
    }
  }]
}

A minimal example of samply profiling output

You’ll notice that this JSON does not include any information about function symbols, so to attribute collected samples to our instrumented functions, we have to map them manually.

We have to start by mapping symbol data to addresses. We can do it by extracting symbol ranges from the program binary. That’s why you have to use a custom profile if you want to profile programs running in a --release mode:

[profile.profiling]
inherits = "release"
debug = true

Otherwise, the symbol data is not in the binary. hotpath uses object crate to extract ranges corresponding to known function symbols.

examples/cpu_nested.rs

#[hotpath::measure]
fn heavy_work(iterations: u32) -> u64 {
    let mut result: u64 = 1;
    for i in 0..iterations {
        result = result.wrapping_mul(black_box(i as u64).wrapping_add(7));
        result ^= result >> 3;
    }
    result
}

#[hotpath::measure]
fn light_work(iterations: u32) -> u64 {
    let mut result: u64 = 0;
    for i in 0..iterations {
        result = result.wrapping_add(black_box(i as u64));
    }
    result
}

#[hotpath::measure]
fn outer_work() -> u64 {
    let mut total: u64 = 0;
    total = total.wrapping_add(heavy_work(500_000));
    total = total.wrapping_add(light_work(100_000));
    total
}

#[hotpath::main]
fn main() {
    let mut total: u64 = 0;
    for _ in 0..1000 {
        total = total.wrapping_add(outer_work());
    }
    black_box(total);
}

It produces the following data structure:

[
    (
        2980,
        3348,
        "cpu_nested::heavy_work",
    ),
    (
        3348,
        3656,
        "cpu_nested::light_work",
    ),
    (
        3656,
        3904,
        "cpu_nested::outer_work",
    ),
]

The JSON profile does not directly say: “this sample belongs to heavy_work“. Instead, it stores sampled stack traces as compact table references.

Here’s how we can resolve the symbol:

samples.stack[0] => 1
stackTable.frame[1] => 1
frameTable.address[1] => 0x0c10 => 3088
symbol_ranges.contains(3088) => cpu_nested::heavy_work

And here’s our leaf function symbol!

Exclusive vs inclusive CPU attribution

For exclusive CPU attribution, this is enough. The sample is attributed to the first instrumented function found in the leaf frame.

hotpath-cpu supports HOTPATH_CPU_INCLUSIVE=1 config, which attributes CPU usage to both instrumented children and parent functions.

Running cpu_nested in a default exclusive CPU attributions mode produces:

cargo run --example cpu_nested --features='hotpath,hotpath-cpu'

+------------------------+---------+---------+
| Function               | Samples | % Total |
+------------------------+---------+---------+
| cpu_nested::heavy_work | 4420575 | 86.44%  |
+------------------------+---------+---------+
| cpu_nested::light_work | 627107  | 12.26%  |
+------------------------+---------+---------+

and in inclusive mode, we get:

HOTPATH_CPU_INCLUSIVE=1 cargo run --example cpu_nested --features='hotpath,hotpath-cpu'

+------------------------+---------+---------+
| Function               | Samples | % Total |
+------------------------+---------+---------+
| cpu_nested::main       | 5235886 | 99.06%  |
+------------------------+---------+---------+
| cpu_nested::outer_work | 5234714 | 99.04%  |
+------------------------+---------+---------+
| cpu_nested::heavy_work | 4515297 | 85.43%  |
+------------------------+---------+---------+
| cpu_nested::light_work | 719417  | 13.61%  |
+------------------------+---------+---------+

These two modes are useful for analyzing how children’s instrumented functions usage relates to their parents.

So how can we attribute CPU traces to non-leaf instrumented functions? The answer is… LINKED LISTS! (╯°□°）╯︵ ┻━┻

Apparently, they do exist outside of LeetCode…

So coming back to our example:

samples.stack[0] => 1
stackTable.frame[1] => 1
frameTable.address[1] => 0x0c10 => 3088
symbol_ranges.contains(3088) => cpu_nested::heavy_work

To find the parent frame, we have to follow the prefix field:

stackTable.prefix[1] => 0

stack node 0:
  stackTable.frame[0] => 0
  frameTable.address[0] => 0x0ea0 => 3744
  symbol_ranges.contains(3744) => cpu_nested::outer_work

stackTable.prefix[0] => null

null means we’ve reached the root of the sampled stack and can stop walking the linked list. This is the core mechanism behind inclusive CPU attribution in hotpath-cpu mode.

I owe linked lists an apology. ┬─┬ノ( º _ ºノ)

CPU delta vs. samples count

So we can now attribute CPU samples to our instrumented functions. Now consider the following example:

examples/cpu_park_vs_busy.rs

const COMPUTE_BATCH: u32 = 200_000;

#[hotpath::measure]
fn busy_compute(iterations: u32) -> u64 {
    let mut result: u64 = 1;
    for i in 0..iterations {
        result = result.wrapping_mul(black_box(i as u64).wrapping_add(7));
        result ^= result >> 3;
    }
    result
}

#[hotpath::measure]
fn park_main(duration: Duration) {
    thread::park_timeout(duration);
}

#[hotpath::main]
fn main() {
    let start = Instant::now();
    while start.elapsed() < Duration::from_secs(5) {
        black_box(busy_compute(COMPUTE_BATCH));
    }

    park_main(Duration::from_secs(5));
}

cargo run --example cpu_park_vs_busy --features='hotpath,hotpath-cpu'

timing - Execution duration of functions.
+--------------------------------+-------+-----------+-----------+---------+---------+
| Function                       | Calls | Avg       | P95       | Total   | % Total |
+--------------------------------+-------+-----------+-----------+---------+---------+
| cpu_park_vs_busy::main         | 1     | 10.22 s   | 10.23 s   | 10.22 s | 100.00% |
+--------------------------------+-------+-----------+-----------+---------+---------+
| cpu_park_vs_busy::park_main    | 1     | 5.00 s    | 5.00 s    | 5.00 s  | 48.95%  |
+--------------------------------+-------+-----------+-----------+---------+---------+
| cpu_park_vs_busy::busy_compute | 16415 | 304.46 µs | 333.57 µs | 5.00 s  | 48.91%  |
+--------------------------------+-------+-----------+-----------+---------+---------+

cpu - CPU sampling attribution per function (exclusive)
+--------------------------------+---------+---------+
| Function                       | Samples | % Total |
+--------------------------------+---------+---------+
| cpu_park_vs_busy::busy_compute | 5044956 | 99.05%  |
+--------------------------------+---------+---------+
| cpu_park_vs_busy::park_main    | 1780    | 0.03%   |
+--------------------------------+---------+---------+

So, in terms of timing, park_main and busy_compute each took ~50% of the processing time. But CPU usage is reported at 99% vs 0.03%. Let’s compare it to the samply UI report:

It reports a 50/50 split similar to the hotpath wall-clock time report. The only hint of CPU usage is a yellow chart at the top of the report.

I’ve considered different approaches and, for now, settled on using threadCPUDelta rather than timing or sample count to calculate CPU attribution %. So the default samply call traces UI attributes samples to a function symbol, even if the CPU is dormant (i.e., thread in parked state) when the sample is collected. Recreating this approach would replicate the wall-clock timing (except for async functions! see sampling comparison for an in-depth info on this topic).

So now hotpath-cpu highlights areas where the CPU really “goes brrr” and augments the report with this new perspective.

Future hotpath releases will likely add more CPU instrumentation modes; feedback on which approach is best is appreciated!

`inline` compiler hints

The current hotpath-cpu implementation uses one more hack to correctly resolve the function symbols:

examples/always_inlined.rs

#[hotpath::measure]
#[inline(always)]
fn always_inlined(iterations: u32) -> u64 {
    let mut result: u64 = 1;
    for i in 0..iterations {
        result = result.wrapping_mul(black_box(i as u64).wrapping_add(7));
        result ^= result >> 3;
    }
    result
}

#[hotpath::main]
fn main() {
    let mut total: u64 = 0;
    for _ in 0..2000 {
        total = total.wrapping_add(always_inlined(50_000));
    }
    black_box(total);
}

heavy_work uses an #[inline(always)] annotation, which causes its address symbol range to disappear from the binary. The current implementation of hotpath::measure macro implicitly strips user-provided inline annotation, and applies inline(never) to prevent this issue. Working on this post, I’ve noticed that samply can still correctly attribute samples even to inlined functions, it even reports in the UI that the function was inlined:

Apparently, samply uses a gimli crate to resolve symbols based on DWARF format. I did not get around to researching this topic yet, but future hotpath releases will likely drop the inline(never) hack and use DWARF format parsing in case symbol ranges are missing from the binary.

See this GH issue if you would like to contribute this improvement.

Summary

A large chunk of my career was dedicated to optimizing Ruby on Rails applications, where numbers below a millisecond rarely even appeared in logs. So working on problems where nanosecond-scale improvements can have a significant impact has completely changed my perspective on performance engineering. Profiling the profiler is fun. I’m looking forward to continuing work on hotpath-rs.

BTW, as much as I enjoy feeding LLMs with my writing, if you’re a human reading this, please leave a comment here or on Reddit so I know you exist.

How to Automate MEV Analysis on EVM Chains using OpenClaw MCP

Sat, 25 Apr 2026 13:38:40 +0200

Searching for MEV profit opportunities often requires tedious blockchain monitoring and analysis. In this tutorial, I’ll describe how to automate this process using the mevlog-rs MCP interface and the OpenClaw agent. We’ll use a secure HTTPS connection and Telegram chat for communication. We’ll also discuss the pros and cons of using OpenClaw vs. a manual MCP integration.

Initial setup and RPC requirements

I don’t think the world needs another article on a basic OpenClaw setup. Any tutorial on running OpenClaw with a Ubuntu VPS should work. I will assume you already have an OpenClaw integration with a communication interface, such as Telegram, configured.

For an MCP integration, you’ll need an LLM agent configured and running locally. For the rest of this article, we will use the Claude Code CLI.

An optional but useful prerequisite for this tutorial is running a Geth full node on the same machine as OpenClaw. Local node offers significantly better performance for scanning larger block ranges. You can also use a premium RPC endpoint, e.g., from Alchemy. But if you want a quick and easy way to test this setup, you can also use the built-in ChainList integration, which automatically selects the fastest free RPC endpoint. But, please remember that free RPC endpoints are often throttled, don’t provide access to older data, and disable EVM tracing features.

Configuring an MCP server with mevlog-rs

Let’s start with a less “magical” setup. We will first configure a local MCP server and later a remote one to enable querying blockchain data by simply talking to your LLM agent. Full reference is in the mevlog-rs MCP docs.

First, install an mevlog-rs CLI with the MCP feature enabled:

cargo install mevlog --features mcp

Start by verifying if basic querying works:

mevlog search -b latest --rpc-url=$ETH_RPC_URL

You should see JSON output with txs from the recent block.

Now run a mevlog MCP server locally:

mevlog mcp --rpc-url=$ETH_RPC_URL

Next, configure it with your Claude Code instance:

claude mcp add --transport http mevlog http://localhost:6671

That’s it! You can query blockchain through an LLM interface by just asking questions like:

query the last 50 blocks for transactions that transferred PEPE (0x6982508145454ce325ddbe47a25d4ec3d2311933) token
find transactions that transferred over 100k USDC (0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48) in the last 200 blocks
which tx transferred the most DAI (0x6b175474e89094c44da98b954eedeac495271d0f) in the last 6 hours

etc.

Claude Code mevlog-rs MCP integration

Configuring a remote MCP endpoint for better RPC performance

The above setup assumes that the mevlog CLI executes on your local machine. For production-like deployments, this is usually not optimal. The largest bottleneck of querying blockchains is downloading the data. mevlog uses caching extensively, so repetitive queries against the same block ranges will be significantly faster, but initial data must be downloaded over the wire.

For the best download performance, I recommend a proprietary full node. But you’re likely not run it in a local network but on an external VPS. To configure mevlog with an external node, you’ll need an authenticated HTTPS gateway using NGINX. So let’s do it now.

I won’t elaborate on configuring SSH access or running a full node on a proprietary VPS. Check my other tutorial for detailed info on how to do it.

To expose your MCP endpoint externally, you’ll need a similar nginx config:

server {
    listen 80;
    listen [::]:80;
    server_name mcphost.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name mcphost.com;

    ssl_certificate /etc/nginx/ssl/mcphost.crt;
    ssl_certificate_key /etc/nginx/ssl/mcphost.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;

    location = /mcp {
        proxy_pass http://127.0.0.1:6671/mcp;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_request_buffering off;
        chunked_transfer_encoding on;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }

    location = /.well-known/oauth-authorization-server {
        proxy_pass http://127.0.0.1:6671/.well-known/oauth-authorization-server;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;
    }

    location = /.well-known/oauth-protected-resource {
        proxy_pass http://127.0.0.1:6671/.well-known/oauth-protected-resource;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;
    }
}

I usually use a Cloudflare proxy and origin certificates to configure a secure SSL connection. Alternatively, you can use Certbot. BTW, if you already have an OpenClaw running on your VPS, you can ask him to configure it; he’ll know what to do.

Now start the authenticated MCP process on a VPS:

MEVLOG_MCP_AUTH_TOKEN=mcp_password mevlog mcp --rpc-url=$ETH_RPC_URL --host=127.0.0.1

and configure your Claude Code to use this external endpoint:

claude mcp add --transport http mevlog-remote https://mcphost.com --header "Authorization: Bearer mcp_password"

and use it as you did before. It’s a simple way to use a high-performance, externally hosted node to query the blockchain from a local LLM agent.

How to query EVM blockchains with the OpenClaw LLM agent

The MCP server required some manual configuration. If you’re more into “agentic workflows”, the OpenClaw setup might work better for you. Once you already have an OpenClaw configured, getting started with mevlog-rs is straightforward. There’s a dedicated SKILL.md file that explains how to use the CLI directly without the MCP endpoint. A recommended setup is to run OpenClaw on the same VPS as your RPC node to ensure the best performance.

Just ask your OpenClaw to import the skill into memory, and provide an RPC (local or remote) endpoint, and you’re good to go.

Query EVM chains by chatting with your OpenClaw agent

I find OpenClaw setup useful for running scheduled queries and reporting directly to a Telegram channel. You can configure cron tasks like:

every 1h check and notify me if any transaction transferred more than 1 million USDC
each day at 8AM send me a report on which 10 txs paid the most for gas in the last 24 hours
how many txs did jaredfromsubway.eth send in the last 24h, and how much was his total gas spending

etc.

Summary

I wish I had access to similar tooling when I was full-time focused on building MEV bots some time ago. It would have saved me dozens of hours of manually querying blockchains for relevant txs. I hope you’ll find some of these techniques useful. Please send any feature requests or bug reports.

Also, stay tuned: I’m currently working on a major release of the mevlog CLI that will massively speed up and improve querying across large block ranges.

Using LLMs and MCP to Debug PostgreSQL Performance in Rails

Tue, 22 Jul 2025 00:09:04 +0200

I’ve recently automated a large portion of my Rails performance audits. In this tutorial, I’ll describe how to configure an AI-powered PG performance debugging. We will cover using LLMs with custom MCP (model context protocol) n8n integration. We will also discuss the legal and security implications of connecting AI to the production database.

How to optimize PostgreSQL performance with AI?

I’m not sure if the current blogosphere needs yet another introduction to MCP. Instead, here’s a sneak peek of what we will be building in this tutorial:

TLDR Slack bot debugging PG performance.

A Slack bot connected to the PostgreSQL database? Can it run arbitrary SQL queries? Does it send production data to OpenAI servers? Is it any good in nontrivial Rails apps? Will it yolo a DROP TABLE on prod at 3 AM?

Let’s answer all these questions.

Debugging context provided by the pg-extras gem

Copy-pasting queries from heroku-pg-extras one weekend turned out to be a good thing. I’ve been maintaining the rails-pg-extras gem for over 6 years now. The project’s relative popularity helped me onboard multiple clients and earn a living as a Rails performance consultant for over four years.

A significant part of my performance audits and subsequent retainer offering was interpreting the output of the pg-extras gem. It currently exposes ~40 methods. A solid entry point is the diagnose method, which aggregates over a dozen checks to produce a report about the database health. By configuring the web interface, you can get a nice dashboard:

rails-pg-extras web dashboard diagnose report

Rails projects with a dynamic development pace and sizeable traffic are likely to discover new db-level inefficiencies every week. But, deep dive into metadata exposed by the pg-extras API can be tedious and repetitive process.

Here’s where a brand new rails-pg-extras-mcp gem makes an entrance.

Automating pg-extras Performance Debugging with LLMs

rails-pg-extras-mcp is a relatively simple (~300 boilerplate LOC total) wrapper on top of rails-pg-extras. It exposes the API in a format digestible by LLMs. By connecting it to your AI model of choice, you can automate the repetitive task of hunting for inefficiencies in the database layer.

This gem is a core component of the previously showcased Slack bot demo. Slack bot is just an arbitrary choice of interface, configured with the help of the n8n platform:

rails-pg-extras MCP with Slack bot n8n schema

MCP protocol is universal, so you can use it with local Cursor IDE, Claude Code, OpenAI API, basically any modern LLM.

Claude Desktop rails-pg-extras MCP integration

I like the Slack bot interface because it works well for team collaboration. Realistically, you cannot expect LLM to resolve PostgreSQL issues with 100% accuracy. However, the usual bottleneck of working with pg-extras is making sense of its verbose output. A raw output is often illegible:

Raw rails-pg-extras output

LLMs do a fantastic job, extracting useful info from raw data and reformatting it to vastly simplify human analysis. I’ve configured the demo Slack bot to only reply if mentioned by the @n8n handle. A development team can discuss ongoing db issues on a dedicated channel and augment context by predigested metadata from the LLM.

LLM-powered PG debugging session

Please remember the Slack bot this is just a choice of interface. You can configure LLMs to email you weekly database healthcheck reports PDF, talk to the database directly from the Cursor IDE chatbox etc.

So it looks pretty cool, but can you use it on production without summoning the GDPR police?

Legal and security concerns of using LLMs with production data

DISCLAIMER: I’m not a lawyer, and this is not legal advice. Please do your own research before exposing any sensitive data to LLMs.

rails-pg-extras core component is ruby-pg-extras gem. It defines metadata queries with performance debug data.

METADATA is an critical keyword here. It means that you can use rails-pg-extras MCP integration without giving LLMs access to ANY business logic-related tables. Additionally, even query argument values are not exposed. Here’s a sample output excerpt of the calls method, which lists top bottlenecks:

SELECT first_name, last_name, email FROM customers WHERE email LIKE $1 ORDER BY last_name LIMIT $2     
SELECT * FROM orders WHERE order_date >= $1 ORDER BY order_date DESC LIMIT $2                          
SELECT * FROM customers WHERE loyalty_points > $1 ORDER BY loyalty_points DESC LIMIT $2

As you can see, values are replaced by $1, $2 placeholders. It means that in the default rails-pg-extras-mcp configuration, LLMs cannot read any sensitive data.

To strengthen this assumption, you can configure a dedicated user with read access only to the metadata tables:

CREATE ROLE extras_viewer NOLOGIN;
CREATE USER extras_user WITH PASSWORD 'your_password';
GRANT extras_viewer TO extras_user;

GRANT CONNECT ON DATABASE your_db_name TO extras_user;
GRANT USAGE ON SCHEMA public TO extras_user;

GRANT SELECT ON pg_stat_statements TO extras_user;
GRANT SELECT ON pg_stat_activity, pg_locks TO extras_user;
GRANT SELECT ON pg_stat_user_indexes, pg_index TO extras_user;
GRANT SELECT ON pg_stat_all_tables, pg_stat_database, pg_settings, pg_namespace TO extras_user;

GRANT EXECUTE ON FUNCTION pg_relation_size(regclass) TO extras_user;
GRANT EXECUTE ON FUNCTION pg_indexes_size(regclass) TO extras_user;
GRANT EXECUTE ON FUNCTION pg_table_size(regclass) TO extras_user;
GRANT EXECUTE ON FUNCTION pg_total_relation_size(regclass) TO extras_user;

and configure the ENV value:

export RAILS_PG_EXTRAS_MCP_DATABASE_URL=postgres://extras_user:your_password@localhost:5432/your_db_name

I’m not sure if it’s enough to keep the GDPR inspectors happy. But, it’s an order of magnitude safer compared to letting LLMs roam free with a full-blown read-write db access.

Enabling optional EXPLAIN ANALYZE mode

If you feel adventurous, you can toggle PG_EXTRAS_MCP_EXPLAIN_ENABLED and PG_EXTRAS_MCP_EXPLAIN_ANALYZE_ENABLED flags. They add explain and explain_analyze tools to the MCP interface:

LLM running EXPLAIN ANALYZE on a bottleneck query

This feature allows LLMs to run EXPLAIN and EXPLAIN ANALYZE on selected SQL queries. Hand-crafting queries for such analysis by replacing $1, $2 placeholders with matching data can be a little tedious and time-consuming. LLMs do a fantastic job automating this process.

A critical difference is that it requires read access to the underlying tables. And while EXPLAIN outputs the only theoretical execution plan, EXPLAIN ANALYZE runs the actual query and reports real execution metrics. It comes with an increased risk of potentially exposing sensitive data to the LLMs. Or even writing to the database if you forget to configure read-only access. I did put some effort into preventing LLMs from injecting malicious SQL.

No hesitation

But I wouldn’t bet that a crafty LLM cannot sneak a DROP TABLE SQL injection past these safeguards. If you have feedback on how this security layer could be improved, please get in touch.

How much does it cost?

So far, I’ve had the most success with OpenAI gpt-4o model. During the two weekends I spent testing the setup, sending ~200 Slack messages, I managed to rack up a whopping total of $1.56 in API charges.

Anthropic models seem significantly more expensive. Claude Sonnet 4 costs ~$0.05 per chat message, and Claude Opus 4 devoured $1 for a single performance analysis. I was unable to distinguish the difference in quality of output depending on the model, but I did not spend much time cross-testing them. An interesting strategy could be to use a pricey deep-thinking model for weekly performance reports, and a cheaper one for everyday chat usage.

On top of that you’ll need an n8n instance for ~$20 month, but you can also self-host it.

Summary

Make sure to check out the rails-pg-extras-mcp readme for info on how to use it with your Rails app. The project in an early POC stage, so contributions are welcome.

A Slack bot, some LLM magic, a few hundred lines of glue code, and just like that… I’ve automated myself out of PG consulting gigs. The future looks scary. I’ll be off now to practice my plumbing skills.

My puts Debugging Workflow in Rails Apps

Mon, 19 May 2025 07:58:21 +0200

Puts debugging is the best. In this tutorial, I’ll walk you through my workflow for debugging Rails codebases using a custom toolkit inspired by Rust. We will also discuss how to smuggle your debug tools, through even the most rigorous code review process.

Why and how to puts debug Ruby code?

I’ve never been a fan of (or properly learned) using debug breakpoints. While I sometimes use ruby debug, most of the time, I’m a puts debugger.

My favorite way to debug is via an automated test suite. I usually have the retest gem running in the background to automatically execute relevant tests on each file save. If I need to narrow the scope, I use a shortcut to copy the current filename with the line number and run a single test:

rs spec/models/user_spec.rb:42

rs is a alias for bundle exec rspec

So far, I’ve relied on quick-and-dirty print statements like these:

p '!!!!!!!!!!!!!!'
p 'first_user', first_user
p 'second_user', second_user
p '!!!!!!!!!!!!!!'

Please don’t tell me you haven’t written something like this before. It highlights a few issues with the default way to debug in Ruby.

!!!!!!!!!!!!!!! - annotation is often needed. p debug statements don’t stand out enough in a verbose test/logs output.

p 'first_user', first_user - manually adding labels is necessary to identify which debug statement is which.

p method calls are hard to clean up later since they might already exist elsewhere in the codebase.

Introducing `dbg-rb` - a simple Ruby puts debugging helper

Recently, I’ve been writing more Rust than Ruby. I really came to appreciate its built-in dbg! macro:

#[derive(Debug)]
struct User {
    username: String,
    password: String,
    active: bool,
}

fn main() {
    let user = User {
        username: "user1".to_string(),
        password: "password1".to_string(),
        active: true,
    };

    dbg!(user);
}

// [src/main.rs:15:5] user = User {
//     username: "user1",
//     password: "password1",
//     active: true,
// }

It produces a verbose, source-located output, together with the name of the variable. Coming back to Ruby, it felt odd that such a simple and handy helper isn’t available out of the box.

Hence, the birth of the dbg-rb gem. It’s my attempt to recreate the Rust dbg! macro in Ruby:

dbg(User.last(2).map(&:as_json)) 

It prints the filename, line number, and source expression — all with colored output for quick scanning. It also works for method and keyword arguments. For me, knowing the source location and seeing labeled variables names makes debugging faster and far more pleasant to work with. What’s more, the dbg method is unique, so it’s trivial to grep and clean up after use.

For more details and configuration options, check out the documentation in the dbg-rb repo.

How to use dbg-rb in Rails without adding it to the Gemfile?

I find dbg-rb useful in my everyday Rails work. But I understand that not all the devs will share this enthusiasm. When doing performance audits, I needed a way to use this tool without cluttering the dependency list — especially in teams that are strict about gem additions.

In theory it’s possible to add dbg-rb to the IRB config to include it in the Rails console context. But it has little value; I rarely debug directly in the console. I use a test suite for that. It means that dbg-rb has to be available in the bundle exec context. But how can you add a gem to the Gemfile without modifying it?

In theory, I could use a local development branch with a custom Gemfile, but cherry-picking changes before submitting a PR is cumbersome.

Here’s a hack that I came up with. It’s similar to my previous post on Rails debug aliases. You can inject dbg-rb into a Rails app via a local initializer file that’s ignored by Git globally.

Start by creating the file with the following contents:

~/.gitignore_global

config/initializers/dbg_rb.rb

Now configure your git to use it:

git config --global core.excludesfile ~/.gitignore_global

Next you can add an inlined lite version of the dbg-rb file to your Rails app initializers folder:

wget https://raw.githubusercontent.com/pawurb/dbg-rb/refs/heads/main/inline/dbg_rb.rb -O config/initializers/dbg_rb.rb

That’s it! You can now use dbg to debug local variables and parameters with automatically appended labels:

dbg user 
# [spec/user_spec.rb:20] user = #<User id: 1, team_id: 1, slack_id: "develop", pseudonym: ...

None of these steps leaves a trace in the git diff, so you can apply it to any Ruby/Rails project without a code review process. Debug tools like this are safe for local use. But be cautious if you’re introducing less obvious techniques — always consult your team before adding anything bypassing standard workflows.

Summary

I find the additional context provided by dbg-rb gem helpful for non-trivial debugging sessions. And if you’re a “vibe debugger”, feeding this extra info to the LLMs increases your chance to quickly produce a correct answer. I wish a similar tool was available by default like in Rust. Unfortunatelly, this Ruby feature proposal has received little traction so far. If you have suggestions for improving dbg-rb — feedback, PRs, and bug reports are always appreciated.

How to Extract long-tail MEV Profit from Uniswap

Tue, 25 Mar 2025 09:29:37 +0100

Any imbalance that makes it past the Mempool is a potential source of MEV profit. In this post, I describe the process of discovering, analyzing, and executing a long-tail MEV strategy on the Ethereum Mainnet. We will discuss Mempool monitoring, oracle backrunning, recurring Defi exploits, and more. I’ll also showcase how to use mevlog.rs CLI and web interface for finding long-tail profit opportunities.

Disclaimer: The information provided in this blog post is for educational purposes only and should not be treated as financial advice.

Spoiler alert: I made $6.50

How to discover and profit from imbalanced ERC20 pools?

Code examples for this tutorial are available in the repo.

The core of the strategy we will explore is called ”skimming”. It’s a mechanism implemented by the UniswapV2 protocol, that is designed to balance token reserves.

UniswapV2 pools have a built-in skimming feature, and MEV bot operators can use it to profit from imbalanced pools.

This is how it works: UniswapV2 pool has two sources of data on its current ERC20 tokens balances. Internally, it keeps reserve0 and reserve1 storage slots, but actual token balances are determined by calling the balanceOf(address) method on the ERC20 token contract.

If balanceOf of token0 returns a value higher than reserve0, a surplus amount can be “skimmed” from the pool or used to make the standard swap operation. This is an expected behavior for a UniswapV2 pool. Any swap operation is executed by first transferring the input token to the pool. Usually, these details are abstracted away by the UniswapV2Router contract. However, MEV bot operators typically use the swap method directly, transferring the input token without the help of the router.

Skimming is arguably one of the simplest MEV strategies:

you don’t need any capital, except for gas fees, to execute it
custom contract is optional; you can call a pool contract directly
off-chain calculations are relatively straightforward, or you can call the skim method without any maths needed

As you can imagine, these opportunities are highly competitive. I’ve run a skimmer bot on a few EVM chains. On side chains, it’s still possible to regularly find a few cents or even 10+ dollars lying around. But the Mainnet is insanely competitive.

I’ve collected a list of ~20 Mainnet UniswapV2-compatible factories with over 400k pools. Here’s a Rust snippet you can use to discover factory contracts by observing Sync(uint112,uint112) logs and some duck-typing Solidity calls:

examples/discover_factories.rs

sol! {
  event Sync(uint112 indexed current, uint112 indexed delta);

  #[sol(rpc)]
  interface IUniV2Pair {
    function factory() returns(address);
  }

  #[sol(rpc)]
  interface IUniV2Factory {
    function allPairsLength() returns(uint);
  }
}

#[tokio::main]
async fn main() -> Result<()> {
    let mut known_factories = HashSet::new();
    println!("Discovering factories...");
    let rpc_url = "wss://ethereum-rpc.publicnode.com";

    let ws = WsConnect::new(rpc_url);
    let provider = ProviderBuilder::new().on_ws(ws).await?;
    let provider = Arc::new(provider);

    let filter = Filter::new()
        .event(Sync::SIGNATURE)
        .from_block(BlockNumberOrTag::Latest);

    let sub = provider.subscribe_logs(&filter).await?;
    let mut stream = sub.into_stream();

    while let Some(log) = stream.next().await {
        let ipair = IUniV2Pair::new(log.address(), provider.clone());

        let factory = match ipair.factory().call().await {
            Ok(factory) => factory._0,
            Err(_e) => {
                continue;
            }
        };

        let ifactory = IUniV2Factory::new(factory, provider.clone());
        let all_pairs_length = match ifactory.allPairsLength().call().await {
            Ok(all_pairs_length) => all_pairs_length._0,
            Err(_e) => {
                continue;
            }
        };

        if all_pairs_length == U256::ZERO {
            continue;
        }

        if known_factories.contains(&factory) {
            continue;
        }

        println!("Found new UniV2 factory: {:?}", factory);
        known_factories.insert(factory);
    }

    Ok(())
}

Running it for a while should produce a similar output:

cargo run --example discover_factories
# Discovering factories...
# Found new UniV2 factory: 0x5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f
# Found new UniV2 factory: 0x5fa0060fcfea35b31f7a5f6025f0ff399b98edf1
# Found new UniV2 factory: 0xc0aee478e3658e2610c5f7a4a2e1777ce9e4f2ac
# Found new UniV2 factory: 0x115934131916c8b277dd010ee02de363c09d037c

It should allow you to quickly gather addresses of active, UniswapV2-compatible factories on any EVM chain. Later, you can get all the pool addresses from the factory contract. Alternatively you can rework this script to analyze past log entries instead of live monitoring.

For my skimmer bot, I’m using aggregate contract similar to UniswapFlashQuery. With this setup running concurrently on a local Reth node, it’s possible to scan these 400k pools for imbalances in ~10 seconds. It was a decent improvement because the initial synchronous and non-batched version took over 15 minutes to scan the same dataset.

It means that all of the new blocks are covered. But, I’ve observed only a few imbalances that made it past the mempool and were immediately snatched by other MEV bots.

Skimming strategy is well-known, so no wonder there was no easy profit there. But I’ve decided to dig deeper.

One exploit a day

One reason for the UniV2 pool to become imbalanced are so-called “rebasing tokens”. These are ERC20 tokens, with balanceOf(address) implementation that does not map directly to the balances storage slot, but has a dynamic component:

A popular token implementing such mechanism is Staked Ether (stETH) by Lido. Here a source code excerpt:

function balanceOf(address _account) external view returns (uint256) {
    return getPooledEthByShares(_sharesOf(_account));
}

function getPooledEthByShares(uint256 _sharesAmount) public view returns (uint256) {
    return _sharesAmount
        .mul(_getTotalPooledEther())
        .div(_getTotalShares());
}

As you can see balanceOf delegates to getPooledEthByShares that multiplies _sharesAmount static storage by results of _getTotalPooledEther() and divides by _getTotalShares(). It means that when outputs of these method calls change, balance of stETH in the UniswapV2 pool will increase (while reserve0 stays the same) making skimming possible.

Call tracing can be useful for quickly spotting ERC20 tokens with non-standard balance logic. Let’s first use mevlog-rs CLI to detect a sample transaction that swapped the stETH and WETH tokens on UniswapV2:

mevlog search -b 100:latest --event 0xae7ab96520de3a18e5e111b5eaab095312d7fe84 --event 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 --event "Sync(uint112,uint112)"

and now visit mevlog.rs to analyze a sample matching transaction call traces:

It proxies output of the cast run command, and since the website uses a Reth archive node, it will work even for older transactions.

You can see that the balanceOf call of the stETH (0xae7ab96520DE3A18E5e111B5EaAb095312D7fE84) contract triggers subsequent calls to the getApp method. Compare it to balanceOf of WETH (0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2), which produces only a simple Return statement. This technique can help detect tokens with nonstandard (potentially rebasing) balance logic without digging through the smart contracts’ source code.

To observe how stETH rebase is extracted, I’ve added a balance monitoring script to investigate what happens when stETH balances increases.

#[tokio::main]
async fn main() -> Result<()> {
    let rpc_url = std::env::var("RPC_WS").expect("RPC_WS is not set");
    let ws = WsConnect::new(rpc_url);
    let provider = ProviderBuilder::new().on_ws(ws).await?;
    let mut block_stream = provider.subscribe_blocks().await?.into_stream();

    let steth = ERC20::new(STETH, provider.clone());
    let mut balance = steth.balanceOf(TARGET).call().await?._0;

    while let Some(block) = block_stream.next().await {
        let new_balance = steth.balanceOf(TARGET).call().await?._0;
        if new_balance != balance {
            balance = new_balance;
            notify_slack(
                &format!("New stETH balance: {}, block: {}", balance, block.number)
            )
            .await?;
        }
    }

    Ok(())
}

After getting the block number, I checked what landed on top by running a mevlog-rs CLI command:

mevlog search -b 21916169 -p 0

As expected, position 0 in this block is a Lido oracle update transaction:

mevlog search -b 21916169 -p 1

Followed by a weird transaction executing the expected skim operation and a dozen “Idol” NFT token transfers. This MEV bot operator is also responsible enough to immediately start farming all the skimmed stETH tokens:

On a side note, the mevlog.rs web version is handy for quickly checking the surrounding context of any transaction. Here’s a link showing data for these two neighboring transactions. On Etherscan, it takes like 5+ clicks to check what’s before and after a transaction from the same block.

You can also get the same data from the CLI:

mevlog tx 0x5afb7ef3d0151ad93d2bd2808b4f333763774635699e52b9846d5cd69abc6403 -a 1 -r --trace revm

It produces:

# Gas Price:         4.82 GWEI
# Gas Tx Cost:       0.00449 ETH | $8.92
# Coinbase Transfer: 0.23117 ETH | $458.65
# Real Tx Cost:      0.23567 ETH | $467.57
# Real Gas Price:    252.75 GWEI

The MEV bot paid ~0.23 ETH bribe to the coinbase miner account and pocketed ~0.026/$55 of profit. After some digging I’ve found that MEV bots apparently incorporated a past Idol NFT token exploit into their strategy. Despite the audit, a vulnerability was discovered around two years after the protocol launch. It allows the owner of a single NFT token to extract all the stETH staking rewards.

The initial attack did not even have to backrun the oracle update or excessively bribe the miner. But it’s almost identical to the recurring MEV extraction that is still taking place. Idol NFTs contract does not use an upgradeable proxy, so this recurring exploit is here to stay.

Before the exploit, the stETH rebase yielded only ~$150 total profit, with ~%98 going for the bribe. Thanks to the Idol NFTs, profit is now significantly higher. It’s awesome to see MEV searchers constantly evolving and learning from each other.

How to find less competitive MEV strategies?

I don’t really want to become an exploitooor for $50, so I kept searching. I returned to the imbalanced UniswapV2 pool that I discovered using the aggregate reserves scanner.

Apparently, WETH/eETH Sushi pool was not skimmed on time, and over $1 of potential profit made it past the Mempool. I’ve determined that exactly like stETH, eeETH is a rebasing token with oracle-based balance updates. One problem with this strategy was that some eETH oracle update transactions were massive at over 4 MILLION gas, making it less likely to sneak them into the Flashbots priority queue. That’s probably the reason why this opportunity made it past the Mempool. I’ve found a single bot account that did the Mempool oracle backrunning for this pair, with the following methods call trace:

This bot operator is using a Sushi router swapExactTokensForTokensSupportingFeeOnTransferTokens method instead of a low-level swap. His transaction uses over 200k gas compared to 120k from my initial simulation. I’ve decided to pursue this opportunity.

Simulating MEV profit opportunity with Foundry

My go-to way to validate an MEV strategy is to write a Forge test/simulation. I’ve started by downloading the eETH token codebase using cast:

cast source 0x17144556fd3424edc8fc8a4c940b2d04936d17eb -d steth

It uses a proxy pattern, so make sure to use the correct address. After digging into the source code, I’ve found a single method that I care about:

Calling it changes token balances, making skimming possible. But it’s restricted to the admin account. Here’s the forge simulation that I’ve implemented:

test/EEthRebase.t.sol

contract EEthRebase is Test {
    IUniswapV2Pair public sushi =
        IUniswapV2Pair(0x6db0fe375ccB8AC3c2a0984D678Bcd99981Ddcb6);
    IERC20 public eeth = IERC20(0x35fA164735182de50811E8e2E824cFb9B6118ac2);
    IERC20 public weth = IERC20(0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2);
    address public ME = address(0xbeef);

    function test_Rebase() public {
        vm.deal(ME, 1 ether);
        uint256 eethBalanceBefore = eeth.balanceOf(address(sushi));
        console.log("Eeth balance before: %d", eethBalanceBefore);
        vm.broadcastRawTransaction(rebase_payload());
        uint256 eethBalanceAfter = eeth.balanceOf(address(sushi));
        uint256 surplus = eethBalanceAfter - eethBalanceBefore;
        (uint112 reserve0, uint112 reserve1, ) = sushi.getReserves();
        console.log("Eeth balance after: %d", eethBalanceAfter);
        console.log("Eeth surplus: %d", surplus);

        uint256 myWethBalanceBefore = weth.balanceOf(ME);
        console.log("My Weth balance before: %d", myWethBalanceBefore);
        uint256 wethAmountOut = getAmountOut(surplus, reserve0, reserve1);
        console.log("wethAmountOut: %d", wethAmountOut);
        sushi.swap(0, wethAmountOut, ME, new bytes(0));
        uint256 myWethBalanceAfter = weth.balanceOf(ME);
        console.log("My Weth balance after: %d", myWethBalanceAfter);
    }

    function getAmountOut(
        uint256 amountIn,
        uint256 reserveIn,
        uint256 reserveOut
    ) public pure returns (uint256 amountOut) {
        uint256 amountInWithFee = amountIn * 997;
        uint256 numerator = amountInWithFee * reserveOut;
        uint256 denominator = reserveIn * 1000 + amountInWithFee;
        amountOut = numerator / denominator;
    }

Running it produces:

forge test -vv --fork-url https://eth.merkle.io --fork-block-number 21986784 --via-ir
# [PASS] test_Rebase() (gas: 177393)
# Logs:
#   Eeth balance before: 16701405221952935511
#   Eeth balance after: 16701674447724070712
#   Eeth surplus: 269225771135201
#   My Weth balance before: 0
#   wethAmountOut: 266528025931141
#   My Weth balance after: 266528025931141

The beauty of simulations is that you don’t have to worry about protocol implementation details. eETH token logic is proxies all the way down, but all I care about is: moar token == better.

The test shows that broadcasting (vm.broadcastRawTransaction) the oracle update transaction changes the pair’s eETH token balances, making the profitable swap operation possible. BTW you can produce a raw, singed payload of any transaction to use for forge broadcasting by running:

cast tx 0x589d4025ac9337f5bfc465b342896be78e978cf5ff6517dd6d3ac60a4869ebef --raw

It was time to set the trap for the next oracle update tx. To do it I’ve configured a similar mempool monitoring script:

example/find_swap_txs.rs

sol! {
  event Swap(address,uint256,uint256,uint256,uint256,address);
}

#[tokio::main]
async fn main() -> Result<()> {
    let rpc_url = std::env::var("RPC_WS").expect("RPC_WS is not set");

    let ws = WsConnect::new(rpc_url);
    let provider = ProviderBuilder::new().on_ws(ws).await?;

    let mut tx_stream = provider
        .subscribe_pending_transactions()
        .await?
        .into_stream()
        .fuse();

    let conf = "{\"tracer\": \"callTracer\", \"tracerConfig\": { \"withLog\": true }}";

    while let Some(tx_hash) = tx_stream.next().await {
        let Ok(Some(tx)) = provider.get_transaction_by_hash(tx_hash).await else {
            continue;
        };
        let tx = tx.into_request();

        let tracer_opts = serde_json::from_str::<GethDebugTracingOptions>(conf).unwrap();
        let tracing_opts = GethDebugTracingCallOptions::default();
        let tracing_opts = tracing_opts.with_tracing_options(tracer_opts);

        let trace = match provider
            .debug_trace_call(
                tx.clone(),
                BlockId::Number(BlockNumberOrTag::Latest),
                tracing_opts,
            )
            .await
        {
            Ok(trace) => trace,
            Err(_e) => {
                // println!("Error tracing tx: {}", e);
                continue;
            }
        };

        let trace = match trace {
            GethTrace::CallTracer(frame) => frame,
            _ => continue,
        };

        let mut all_calls = Vec::new();
        collect_calls(&trace, &mut all_calls);
        for trace in all_calls {
            let logs = trace.logs;

            for log in logs {
                let Some(topics) = log.topics else {
                    continue;
                };

                if topics.is_empty() {
                    continue;
                }
                let first_topic = topics[0];
                if first_topic == Swap::SIGNATURE_HASH {
                    println!("Found swap tx {}", &tx_hash);
                }
            }
        }
    }

    Ok(())
}

fn collect_calls(frame: &CallFrame, result: &mut Vec<CallFrame>) {
    result.push(frame.clone());

    for call in &frame.calls {
        collect_calls(call, result);
    }
}

Running it produces:

cargo run --example find_swap_txs
# Found swap tx 0xb9e154a84948889b93accb1d08e15b816dd92f111ce7e8f6d9df0bfbbeba0f8c
# Found swap tx 0xa8bb148a32e0c4bb37b2dda6c08f8c1aa3ad8fadc28725eaed5a09885a8fe910

You’ll need a WebSockets RPC endpoint with debug RPC methods enabled to run this script. It monitors the mempool and traces all the discovered transactions using the built-in {tracer: 'callTracer'}. Later, it extracts all the logs and prints tx hashes of transactions that emit matching logs.

To execute the eETH oracle backrun I’m monitoring for the Rebase(uint256,uint256) event generated by a transaction originating from the eETH admin account. It eliminates the need to trace every single transaction in the mempool. But please remember that for mempool monitoring, you’ll probably need a proprietary full node or a commercial 3rd party plan. This mode of operation is likely to exhaust free RPC node limits quickly.

Once you get a matching tx hash, the rest of the process goes as follows:

use get_raw_transaction_by_hash method to get a signed transaction payload
spawn a local Anvil process forking of the mainnet state
change the local fork state by submitting signed transaction using the send_raw_transaction method
calculate a profitable swap amount and prepare your skimming transaction payload
prepare a Flashbots-compatible bundle with oracle update and swap tx and submit it to the builders using eth_sendBundle RPC call

Describing all these steps with code examples would bloat the size of this post. But let me know in the comments if you’d like to read a more detailed tutorial on executing this strategy.

Step three is profit

Contrary to my previous post about Revm arbitrage simulations this time I have some actual profit to show for. Here I am, doxxing myself for the internet points. My first eETH oracle backrunning transaction made ~$0.35 of profit!

For a few days after submitting the first successful oracle backrun, I was dominating this corner of the blockchain and earned a total of ~$6.50:

Later, I shifted my focus to releasing the web version of mevlog.rs. After migrating from the Geth full node to the Reth archive node, I’ve only recently fixed the tracing config bug. So, maybe, this strategy will print a few more $ before the end of the month.

Summary

Finding any loophole in the current state of the Mainnet MEV game is always fun. I hope some of the described tools and techniques will prove helpful for your daily MEV search. And maybe help discover opportunities that can buy more than a cup of coffee. I’m up for more mempool spelunking in the near future so don’t forget to like and subscribe to get notified about new posts.

How to Discover long-tail MEV Strategies using Revm

Mon, 17 Mar 2025 06:29:37 +0100

It’s extremely challenging for MEV newcomers to profit from popular strategies. That’s why, I’ve refined my searching approach to focus on niche and less obvious opportunities. In this blog post, I’ll showcase a Rust CLI tool that helps me query blockchain data to discover long-tail MEV. I will also describe Revm vs RPC EVM tracing techniques and their performance characteristics.

Disclaimer: The information provided in this blog post is for educational purposes only and should not be treated as financial advice.

How to monitor EVM chains for MEV opportunities?

Searching for long-tail MEV requires investigating and analyzing nonobvious transaction details.

Popular tools like libmev.com or eigenphi.io focus on short-tail MEV like arbitrage and liquidations. It’s possible to extract all the transaction data from Etherscan, but I find it cumbersome to click around web UIs. It takes over five clicks to get neighboring (before/after) tx info… Also, please tell me why does Etherscan display block transactions in reverse order? Isn’t the top of the block always the most interesting??

(╯°□°）╯︵ ┻━┻

I’ve found myself writing one-off scripts parsing blockchain data to extract relevant info. So, I’ve decided to develop them into a simple to use CLI for querying blockchain with configurable EVM tracing and filtering options.

mevlog-rs CLI is inspired by cryo, mev-inspect-py, and cast run. It’s a command line tool that outputs EVM transaction details in a visual and quick-to-digest format:

mevlog-rs displaying one of the infamous sandwiches by jaredfromsubway.eth

I wanted to build a tool that makes it simple to query and lookup transactions using publicly accessible endpoints. If anything catches your interest you can use the more advanced tooling via clickable links to investigate further.

By using a local openchain.xyz signatures database mevlog displays human readable data instead of hex blobs. Root method calls and events provide an instant overview of what the transaction is all about. Thanks to EVM tracing capabilities, mevlog tracks how much bribe was paid to the coinbase and calculates the real effective gas price (i.e. (gas_price + bribe) / gas_used).

You can also filter transactions by regular expressions matching emitted event signatures and addresses.

Another feature is detecting tx storage changes, allowing to filter txs by smart contract addresses/ERC 20 tokens that were affected. There’s also ENS domains integration.

A few examples of currently supported queries:

find jaredfromsubway.eth transactions from the last 20 blocks that landed in positions 0-5:

mevlog search -b 20:latest -p 0:5 --from jaredfromsubway.eth

unknown method signature contract call in top position (likely an MEV bot):

mevlog search -b 10:latest --method "<Unknown>" -p 0

query last 50 blocks for transactions in the top 10 slots that transferred PEPE token:

mevlog search -b 50:latest -p 0:10 --event "Transfer(address,address,uint256)|0x6982508145454ce325ddbe47a25d4ec3d2311933"

blocks between 22034300 and 22034320, position 0 transaction that did not emit any Swap event:

mevlog search -b 22034300:22034320 -p 0 --not-event "/(Swap).+/"

search blocks range for events containing rebase and Transfer keywords:

mevlog search -b 22045400:22045420 --event "/(?i)(rebase).+/" --event "/(Transfer).+/"

All the above queries use only standard block and logs input. It means that you can use publicly available RPC endpoints like https://eth.merkle.io and it should not get rate limited. These queries don’t need state data to execute so it works even for very old blocks on non-archive nodes:

mevlog search -b 14061012 --rpc-url https://eth.merkle.io

Querying over 3yo block data on a public non-archive node

Revm-powered EVM tracing

By enabling --trace [rpc|revm] flag you can query by more conditions:

query last 5 blocks for top transactions that paid over 0.002 ETH total (including coinbase bribe) transaction cost:

mevlog search -b 5:latest -p 0 --real-tx-cost ge20000000000000000 --trace revm

A sample matching tx:

It shows how important it is to track coinbase transfers when analyzing tx costs. This tx paid only $0.33 in standard gas price but landed at the top on the block by bribing the validator over $11k.

You can also filter by real per gas price:

mevlog search -b 5:latest -p 0:5 --real-gas-price ge10000000000 --trace revm

It will display all the txs that paid more than 5 gwei of gas price per unit.

Another example of EVM tracing powered query:

find txs that changed storage slots of the Balancer vault contract:

mevlog search -b 10:latest --touching 0xba12222222228d8ba445958a75a0704d566bf2c8 --trace rpc

It uses EVM state tracing to match contracts whose state was affected by transactions. It’s more general purpose than event-based filtering.

How to analyze MEV sandwich attacks?

mevlog tx method let’s you investigate a single transaction. A grep-like querying API allows to check what was the surrounding (potential sandwich) context. For example, after finding this jared tx in position 1 you can easily check what was the transaction that it backrunned:

mevlog tx -b 1 0x2aaf037240be70f05468c2c6d2e3fe948aea11783c3f90a2237c0c6978b6667b

In this particular example, it looks like it’s an rETH token oracle update.

If you want to investigate sandwich attacks by jared you can start from the following query:

mevlog search -b 20:latest -p 2 --from jaredfromsubway.eth

The lower “bun” of a sandwich is usually in the 2nd position in a block. After grabbing the tx hash you can see preceeding txs like this:

mevlog tx -B 2 0x29c93f92350f3dac6e729ccc5139345128b168ac15c0212854d171b8606a6358

It looks like this poor swap tx did indeed fall prey to jared.

There’s also a live monitoring mode supporting the same query methods as filter:

mevlog watch -p 0:5

I invite you to check out the repo for a full breakdown of the currently available filtering methods. You can start using the CLI by running:

git clone https://github.com/pawurb/mevlog-rs
cd mevlog-rs
cargo install --path .

or install from the crates.io:

cargo install mevlog

One installed you can run:

mevlog watch --rpc-url https://eth.merkle.io

I could not find a CLI tool with similar features. But please let me know in the comments if I’m reinventing the wheel.

Read on if you’re interested in the performance, usability, and EVM tracing-related challenges I’ve faced when building the MVP.

SQLite and ENS integration

Mevlog uses a local openchain.xyz signatures database. Compressed (~120MB) SQLite database file with ~800k event and over 2 million method signatures is downloaded on the first run. It allows to display human-readable info about emitted events and root method calls.

This is my first time using SQLite with this amount of data in a single table. It works seamlessly, fetching method signatures from the table with over 2 million rows in ~50-200µs (microseconds) per query. That’s 5-20 queries per millisecond! I’ve never seen similar numbers when working with PostgreSQL. Relying on locally cached SQLite enables the event and method name regexp search without using Etherscan API calls or parsing ABI files. Uncompressed database file currently weights ~260MB.

ENS integration was another interesting challenge. Initial implementation did 3 RPC calls to get the hashed node address, resolver, and eventually the name, causing ~400ms overhead per name lookup. For ~100 tx per block that’s 40s overhead. By moving node hash calculation off-chain and aggregating two remaining calls with a custom contract I’ve managed to reduce synchronous overhead to ~100ms.

Another optimization was delegating ENS resolutions to the cacache-rs file cache in the background thread. With these changes displaying ENS names has virtually no performance overhead.

Optimal EVM tracing strategies with Revm and RPC

However the most significant implementation challenge was adding the EVM tracing. We need to know how much ETH is bribed to the coinbase to calculate the real gas price. To do it, we need to extract an internal subtransaction trace with empty calldata and to set at the current coinbase.

This implementation is relatively simple for nodes that support the debug_traceTransaction RPC method. It’s possible to recursively extract subtraces output from the built-in {tracer: 'callTracer'} until there’s a match. But I wanted mevlog to also work with free nodes that don’t expose the debug APIs.

From my previous project, I knew that Revm supports tracing capabilities via the revm-inspectors crate. Actually, that’s what foundry uses under the hood for cast run 0x123... to print these detailed call traces:

"Executing previous transactions from the block" output is critical here. Contrary to debug_traceTransaction, Revm does not know the exact state of transactions further down the block, so to trace tx at slot 10 it has to first execute and commit the 9 preceeding txs. There’s a --quick flag that ignores previous state changes, but using it would fail most of the transactions.

I did not manage to wrap my head around the low-level Revm forking implementation of cast run. Instead I’ve used kind of a shortcut. To trace transactions from block N, mevlog spawns the Anvil process to fork off the N-1 block and uses Anvil provider with the Revm SharedBackend. Later, it passes the block N context to the Revm simulation engine and sequentially executes and traces all the transactions.

Let’s compare the performance of tracing the same transaction in the 10th slot with mevlog and cast run:

time mevlog tx 0x2884d487f3fa0c5d983abb415a25c3be0982b50d4107bdabead1498b652f38ad --trace revm
# 0.10s user 0.05s system 1% cpu 9.462 total

time cast run 0x2884d487f3fa0c5d983abb415a25c3be0982b50d4107bdabead1498b652f38ad
# 0.57s user 0.09s system 7% cpu 8.213 total

I’ve measured it against an external RPC endpoint, so any difference in performance is probably due to networking conditions. It indicates that despite the N-1 Anvil forking shenanigans mevlog --trace revm seems to be working correctly.

It’s worth noting that both commands rely on the foundry-fork-db crate. So subsequent traces against the same transactions will be much faster. In this case, it is down to ~1s. foundry-fork-db caches all the relevant data slots in a local file cache. Let’s see how much data is needed to execute the previously mentioned transaction:

RUST_LOG=debug cast run 0x2884d487f3fa0c5d983abb415a25c3be0982b50d4107bdabead1498b652f38ad

Based on the cast logs output, without cached data this command triggers over 200 RPC calls to the origin. That’s the price for running local Revm simulations because all the relevant storage slots must first be downloaded.

Mevlog also supports the --trace rpc mode that relies on the debug_traceTransaction method. It’s not available on free RPC endpoints, but it can trace transactions over 3rd party RPC in less than a second without caching. Revm-based simulations are great to compare different scenarios on the same storage dataset. But for one-off simulations (and especially for txs further down the block), RPC tracing will usually deliver a better performance.

For example, the first Revm tracing of top 50 txs from a sample block:

mevlog search -b 22053318 -p 0:50 --trace revm  
# 0.75s user 0.43s system 4% cpu 24.437 total

second (with cache):

mevlog search -b 22053318 -p 0:50 --trace revm  
# 0.21s user 0.08s system 10% cpu 2.618 total

and using RPC tracing:

mevlog search -b 22053318 -p 0:50 --trace rpc  
# 0.10s user 0.09s system 2% cpu 8.260 total

and no tracing (i.e. no access to coinbase bribe data):

mevlog search -b 22053318 -p 0:50 
# 0.06s user 0.05s system 6% cpu 1.749 total

Future plans

mevlog is currently in an MVP phase, and many implementation details are far from optimal. Let’s see how long it takes to query the last 1000 blocks for matching event names on a local Geth node:

time mevlog search -b 1000:latest --event "Sync(uint112,uin112)" 
# real    2m55.397s
# user    1m28.724s
# sys     1m12.775s

and last 250 blocks with revm tracing:

time mevlog search -b 250:latest --event "Sync(uint112,uin112)" --trace revm
# real    3m24.163s
# user    0m46.712s
# sys     0m26.668s

In its current state, mevlog is only useful for small block ranges querying and live monitoring. For example, the current search implementation is terribly inneficient fetching logs and transactions data block by block without any batching or concurrency.

For future releases I’m considering to piggyback on the cryo crate for data extraction. Let’s compare the numbers for fetching the last 1000 blocks data:

cryo blocks logs transactions --blocks -1000:latest
# collection summary
# ──────────────────
# - total duration: 6.556 seconds
# - total chunks: 2
#     - chunks errored:   0 / 2 (0.0%)
#     - chunks skipped:   0 / 2 (0.0%)
#     - chunks collected: 2 / 2 (100.0%)
# - blocks collected: 1,000
#     - blocks per second:     152.5
#     - blocks per minute:   9,151.0
#     - blocks per hour:   549,062.7
#     - blocks per day: 13,177,503.7
# - rows written: 552,669

That’s a decent ~98% speedup. With this setup, mevlog could provide a performant, MEV-focused querying API on top of cryo.

Summary

mevlog-rs is in the beta stage so any feedback would be appreciated. I’m just starting to use it for my daily searching tasks. But it already helped me discover and execute a long-tail MEV strategy on the Mainnet. Next week I’m planning to release a detailed post describing the process, so stay tuned for updates.

Lessons Learned Migrating my SAAS to Rails 8

Tue, 14 Jan 2025 08:29:37 +0100

My side project Abot for Slack, has been around for ~7 years, i.e., since Rails 5.1. It’s now yielding passive income while running on an almost complete autopilot. But I’m still keeping its dependencies up-to-date. The recent migration to Rails 8 was arguably the most impactful in the app’s lifetime. In this blog post, I’ll describe features introduced in the newest version of the framework and how they affected my project.

Kamal vs Dokku

TLDR initial impressions migrating Abot from Dokku to Kamal: it’s better, apart from one major flaw.

Abot started as a weekend project deployed to Heroku free dynos. Later, I migrated it to AWS EC2 with the RDS PostgreSQL database using Dokku.

The initial Dokku setup long long time ago

Dokku requires a bit more dev-ops expertise than PAAS solutions, but after the initial setup, it offers a workflow almost identical to Heroku. I’ve used Dokku for over 4 years, and it provided a seamless deployment experience with very few glitches along the way. This is one of the OS projects that I regularly donate to.

But the times are changing, and Kamal is a fancy new tool that everyone is talking about.

Kamal’s great advantage is a declarative one-file config, providing an instant overview of the current state of the project. With Dokku, similar insight is scattered across several commands.

Another benefit is seamless deployment setup. Just point Kamal to a barebones VPS, and everything gets deployed via a single kamal setup command. Dokku requires manual installation on the server unless configured with a custom Ansible playbook or similar tool.

Yet another win for Kamal is support for multi-server architecture. With Dokku, you’re limited to single server deployments, so you have to fit all the services, including databases, caches, etc., on a single VPS.

One thing that’s missing from Kamal is the built-in Nginx integration. Dokku provides a customizable Nginx template allowing for fine-tuned configuration of headers, SSL, compression, etc. With Kamal, you’d need to manually configure Nginx in front of the proxy to achieve the same effect. Apparently, the thruster gem can provide similar features as Nginx, but I have not explored it yet. In the end I’ve reimplemented each of the Nginx config tweaks directly in Rails or Cloudflare.

But here comes the previously mentioned major flaw.

Kamal registry-based deployments

Dokku uses a git-based deployment flow, i.e. each git push triggers a new docker image build combining the base image with app-specific Dockerfile. A new docker image is built directly on the target VPS and kept locally.

Kamal requires a nonoptional registry.server (defaults to "hub.docker.com"), registry.username, registry.password config values. These are credentials for one of the commercial (DockerHub, GitHub, Digital Ocean) or custom docker registry service. It’s possible to spin up your own registry, but it’s cumbersome. Built images are pushed to the registry and later downloaded by a target host.

An issue is that the default Kamal configuration exposes Rails application source code in a public image hosted on Docker Hub. In theory, it could leak nonpublic projects if someone followed the Kamal tutorial without analyzing each step. I wish the Kamal team was more explicit about this behavior in the docs and learning materials.

A way to prevent exposing project source code is manually creating the private image up-front or changing the default visibility in DockerHub settings:

DockerHub image privacy settings

A free DockerHub account can host only a single private image. Even the $11/month plan offers only 5GB storage, so with image sizes at ~1GB it might not be enough. A $16/month offer with 50GB space for private images could be necessary for many users. It might not seem expensive. But a choice between exposing source code or incurring monthly costs could be discouraging for someone just exploring the Rails stack.

Fortunately, there are rumors about the Kamal team working on a way to deploy containers, without relying on 3rd party commercial services. In my opinion, this change would significantly enhance Kamal’s appeal.

Goodbye Redis

Since I started working in Rails around a decade ago, the Redis + Sidekiq combo was the de-facto standard for most projects. Rails 8 challenges this status quo by defaulting to the solid_queue gem for background processing. Solid Queue uses SQL database storage instead of Redis. Together with mission_control-jobs UI it’s a drop-in replacement for Sidekiq. It means that we can now get rid of another process and dependency to maintain.

Caching was another area where I’ve seen widespread use of Redis. On the side node, reusing the same Redis instance for both Sidekiq and caching has always been a fun source of sneaky bugs when both processes’ data interfere despite the namespaces…

(╯°□°)╯︵ ┻━┻

Past traumas aside, with the introduction of solid_cache gem modern Rails apps can remove Redis from their stack, simplifying the infrastructure. While conducting my Rails performance audits, I’ve seen projects where Redis was responsible for a significant portion of monthly infra costs. Migrating to SQL-based solutions can result in a major cost reduction for many Rails projects. Also, as per Rails 8 marketing: “you can now cache weeks instead of hours worth of data”.

Here’s a breakdown of monthly AWS cost for ElastiCache Redis with a specified RAM vs RDS database disk space:

Size	Redis RAM	RDS disk space
1.3GB	~$23	~$0.16
6.3GB	~$110	~$0.78
52GB	~$900	~$6.5
210GB	~$3300	~$26

Size values were chosen to find matching ElastiCache instance

RDS incurs additional db instance costs, but on a larger, scale they will be insignificant. Cache-specific databases execute only simple queries over well-indexed tables, i.e. no joins or seq scans. That’s why caching database CPU/RAM can usually be much smaller than primary. But, workload and traffic patterns differ for each project, so please run your benchmarks before making any production infra changes. More cost-aware projects can also squeeze both databases on a single RDS instance.

I did calculations for ElastiCache, so you could slightly reduce the cost by running Redis on EC2 instances. But RAM will always be an order of magnitude more expensive than disk space.

I’ve seen comments that migrating to “non-memory” cache storage will result in a performance drop. So let’s see the numbers in action.

Measuring non-memory caching performance

I’ve run the following benchmark script against different Rails cache stores:

keys = []
json = File.read("sample.json")
Benchmark.bm do |x|
  x.report("cache") do
    2000.times do |i|
      if i % 5 == 0
        key = "test-#{i}"
        Rails.cache.write(key, json)
        keys << key
      end

      Rails.cache.read(keys.sample)
    end
  end
end

Please take this benchmark with a grain of salt. It’s a straightforward script that does not simulate how the cache store behaves under a constant load or when there’s more data.

It distributes read and write operations in 80/20 relation. Here’s the result for different store types:

Store	Total time	Avg. time
FileStore EBS	20.0s	~4ms
Redis/Memcache	20.1s	~4ms
FileStore EFS	22.4s	~4.5ms
SolidCache SQLite	24.4s	~4.8ms
SolidCache PG	27.05s	~5.4ms

For me, the main takeaway from the above table is that in the context of web applications, the caching layer is fast enough, regardless of the underlying store type. The amount of data that can be cached is vastly more important than sub-millisecond performance improvements.

Although the in-memory Redis/Memcache should be the fastest, they are on par with the File Store EBS. The reason is probably the lack of networking overhead for local file storage.

Another interesting observation is that the FileStore is consistently ~10-20% faster than SolidCache SQLite. It makes sense as SQLite is a layer of abstraction on top of the file system, so it incurs additional overhead. I’m not 100% sure if this metric will hold true for systems under production-like load. But if you want to squeeze out a few more microseconds then the the good ol’ ActiveSupport::Cache::FileStore instead of SQLite could be a better choice.

Please remember that AWS EBS (even gp3) volumes are notoriously slow. When running the benchmark locally on an MBP, I’m observing ~3x better performance for FileStore and SolidCache SQLite. So if you want to maximize the SQLite and cache performance, choosing a VPS provider with high-speed NVMe disks is probably worth it.

While we’re at data stores, I would like to highlight the AWS EFS file system. It’s EBS-equivalent but can be shared between multiple EC2 instances. It’s ~10x more expensive than EBS, but using it instead of PostgreSQL would mean saving money on database instances. It could be an interesting choice for projects using SQLite with multi-server architecture or for leveraging a shared file-based caching layer. I’m currently working on a blog post covering more details on using EFS with Rails. You can follow or subscribe to get notified when it’s live.

Using a custom compressor for Rails cache

Another exciting feature introduced in Rails 8 is support for custom cache compressors. I’ve built the rails-brotli-cache gem around this idea, so it’s awesome to see it upstream. The feature has not made it to the Rails guides yet, but hopefully, this PR will get merged sooner or later. As of now compressor config is described only in ActiveSupport::Cache::Store docs.

I’ve tested a few compression algorithms when working on rails-brotli-cache (benchmark). Choosing “the best one” is a matter of tradeoffs between performance and compression ratio. But zstd-ruby with level: 10 config is a quick & easy choice, offering speed and compression superior to both gzip and brotli.

lib/zstd_compressor.rb

require "zstd-ruby"

module ZSTDCompressor
  def self.deflate(payload)
    ::Zstd.compress(payload, level: 10)
  end

  def self.inflate(payload)
    if payload.start_with?("\x78")
      Zlib.inflate(payload)
    else
      ::Zstd.decompress(payload)
    end
  end
end

config/production.rb

config.cache_store = :solid_cache_store, { compressor: ZSTDCompressor }

Applying this config should improve caching read/write performance by ~30% and compression ratio by ~20%. If you want to use this feature make sure to include:

config.load_defaults "7.1"

in config/application.rb. This change is backward compatible, so you can start using the new algorithm without clearing the current cache contents.

SQLite vs PostgreSQL

In Rails 8 SQLite becomes a first-class citizen. This talk by Stephen Margheim provides an in-depth look at the changes introduced and explains how SQLite’s support and performance improved.

I briefly considered switching Abot’s primary database to SQLite instead of PostgreSQL. I’d love to cut the monthly RDS costs, but I’m not comfortable keeping the business-critical data on the server. “cattle not pets” they said. I’m used to regularly nuking and reprovisioning EC2 instances, so the chances of accidentally wiping production data skyrocket with SQLite.

litestream is an awesome tool for continuously backing up SQLite. But I’d rather not bet the project’s existence on it working seamlessly.

Another disadvantage of SQLite vs. PG is the lack of advanced monitoring. Tools like PgHero, (shameless plug alert!) rails-pg-extras, or pg-locks-monitor provide detailed insights into what’s going under the hood in PostgreSQL.

With SQLite going mainstream in the Rails world, hopefully, open source will soon catch up, and we’ll have more ways to analyze it. I’ve started building rails-sqlite-extras. But I don’t have much SQLite exp, so ideas on how to improve it are welcome.

For now, I’m happy to keep Abot’s solid queue and cache databases on SQLite and primary on PostgreSQL.

Summary

Rails 8 is a decent step forward. It makes things cheaper, faster, more productive. There are some drawbacks, like the Kamal mandatory registry, but I hope it will be sorted out soon.

How to Send deadbeef Transactions with Alloy

Wed, 11 Dec 2024 09:59:17 +0100

0x0000000 accounts are so last season. Now it’s all about vanity tx prefixes. In this blog post, we will learn how to impress your Etherscan followers and fry a few CPU cores along the way.

Vanity txs

It all started from this X thread. Apparently, there’s a bigbrainchad.eth bot, that extracts the MEV in style. All his txs hashes start with the 0xbeef prefix:

I immediately got jealous and also wanted to send a few cool-looking txs. Hence the birth of alloy-deadbeef lib, and soon my vanity tx landed on the mainnet:

Read on to learn about the lib’s implementation details and performance overhead of sending such fancy txs.

How to brute-force ETH tx prefix?

You can use alloy-deadbeef as an alloy filler:

let provider = ProviderBuilder::new()
    .filler(DeadbeefFiller::new(
        "beef".to_string(),
        wallet.clone(),
    )
    .wallet(wallet)
    .on_http(endpoint().parse()?);

All the txs sent from this provider will land with a 0xbeef prefix.

Or generate a tx object that will commit to the chain with the desired prefix (unless you change any of its data!):

let tx = TransactionRequest {
    from: Some(account),
    to: Some(account.into()),
    ..Default::default()
};

let deadbeef = DeadbeefFiller::new(
    "beef".to_string(),
    wallet
);

let prefixed_tx = deadbeef.prefixed_tx(tx);

The lib brute forces a matching hash prefix, by incrementing the transaction gas_limit or value attribute. More details on that later.

We’re working with 16^n complexity here. Even Rust takes a while to crunch these numbers. To optimize performance, the lib is maxing out parallelism by spanning tokio task for each CPU available. The core loop looks like this:

src/lib.rs

let mut buf = Vec::with_capacity(200);

let result: Option<U256> = loop {
    select! {
        biased;
        _ = done.recv() => {
            break None;
        }
        _ = futures::future::ready(()) => {
            let tx = tx.clone();
            let next_value = tx.value.unwrap_or_default() + U256::from(value);
            let tx_hash = tx_hash_for_value(tx, &wallet, next_value, &mut buf).await?;
            value += 1;

            let hash_str = format!("{:x}", &tx_hash);
            if hash_str.starts_with(&prefix) {
                let iters = value - starting_input;
                let total_iters = max_cores * iters;
                info!("Found matching tx hash: {tx_hash} after ~{total_iters} iterations");
                break Some(next_value);
            }
        }
    }
};

async fn tx_hash_for_value(
    tx: TransactionRequest,
    wallet: &EthereumWallet,
    value: U256,
    buf: &mut Vec<u8>,
) -> Result<FixedBytes<32>> {
    let mut tx = tx;
    buf.clear();
    tx.value = Some(value);
    let tx_envelope = tx.build(&wallet).await?;
    tx_envelope.encode_2718(buf);
    let tx_hash = keccak256(&buf);
    Ok(tx_hash)
}

Tx hash is generated by running keccak256 on a signed transaction body, which is why we need to provide an EthereumWallet object. We use a select! macro to kill the remaining loops when the correct prefix is found.

Initially, I’ve used Arc<Mutex<bool>> like this:

  let is_done = done.lock().await;
  if *is_done {
      break None;
  }
  drop(is_done);

But multiple threads competing for the mutex lock on each loop did wreck the performance. It was ~6x slower than tokio CancellationToken:

In the end, I’ve switched to tokio::sync::broadcast::channel. Contrary to CancellationToken, it does not require a separate crate, and is just as fast.

Another interesting discovery was that tokio::time::sleep(Duration::from_millis(0)) has a delay of ~1ms. It killed the performance until I replaced it with futures::future::ready(()), which is instant.

Benchmarking tx prefix generation

I wanted to include a benchmark showing how long on avg. it takes to find a matching prefix. But I’ve found out it’s very random. 0xbeef can take anywhere between 200ms and 900ms, depending on the rest of the transaction attributes. Instead, I’ve AI-ed my way to finding how many iterations are needed to produce the desired prefix with 99% probability:

Length	Probability (1 in)	Iterations for ~99% certainty
1	1/16	~72
2	1/256	~1,180
3	1/4,096	~18,900
4	1/65,536	~302,000
5	1/1,048,576	~4,830,000
6	1/16,777,216	~77,900,000
7	1/268,435,456	~1,240,000,000
8	1/4,294,967,296	~19,800,000,000
9	1/68,719,476,736	~316,000,000,000

An interesting takeaway is that increasing the gas limit by up to 300,000 should not cause any issues with landing the transaction on-chain. The current implementation of alloy-deadbeef iterates on gas_limit instead of value for prefixes up to 4 characters. It means that we can also flex for non-payable transactions:

Alternatively, you can force a specific iteration mode like this:

let mut deadbeef = DeadbeefFiller::new("beef".to_string(), wallet)?;
deadbeef.set_iteration_mode(IterationMode::Value);

Let’s now analyze the maximum processing time. Assume we’re using a computer with 4 CPU cores and searching for 4-character prefix. Each loop has to process a maximum of 302,000 / 4 i.e. 75,500 iterations. So iteration starts from the following values for each of the loops:

0 - 0
1 - 75500
2 - 151000
3 - 256500

The table contains info on what’s the maximum processing time on my MBP M2 and beefed out 48 vCPUs Hetzner:

Length	MBP M2 12 CPUs	Hetzner VPS 48 vCPUs
1	547.58µs	397.69µs
2	5.81ms	3.18ms
3	123.86ms	45.38ms
4	1.82s	755.83ms
5	29.62s	12.08s
6	~8 minutes*	~3 minutes
7	~128 minutes*	~48 minutes*
8	~34 hours*	~13 hours*
9	~23 days*	~8 days*

* extrapolated

Hetzner VPS maxed out crunching 0xdeadbeef

You can see that each additional prefix length increases processing time by ~16x, just as we’ve calculated. But these are the worst-case numbers.

My 0xdeadbeef tx in practice took ~1.5 hour and slightly over 2 billion iterations to generate. According to my AI friends, the probability of this happening was at ~37%.

Summary

Our research shows that it is possible to efficiently use up to 4 character-long prefixes. It could be a simple way to intimidate your MEV competition and assert alpha bot dominance. alloy-deadbeef was a fun weekend project. I’d love to learn how to optimize it further, so please send over any suggestions/PRs.

How to use Cloudflare Workers proxy with Rust

Wed, 02 Oct 2024 09:56:01 +0200

Visits counter was a critical feature of every website just 20 years ago. In this tutorial, we will implement it with Rust Cloudflare Workers by adding persistence and dynamic behaviors to an otherwise static page. We will also discuss other practical use cases of CF workers edge proxy.

Static blog with CF edge caching

Visits: [VISITS_COUNT]

This blog is a static JekyllRB website hosted on an EC2 behind an NGINX proxy. Additionally, it’s using a Cache everything Cloudflare cache rule with the following header for each HTML page:

cache-control: public, max-age=3600

You can check it by running this cURL:

curl -I https://pawelurbanek.com/cloudflare-workers-rust | grep cache

# cache-control: public, max-age=3600
# cf-cache-status: HIT

cf-cache-status: HIT indicates that the request did not reach the origin server but downloaded the HTML page from the CF edge cache. It ensures the best loading performance because edge locations are always close to the readers.

Each HTML page is cached for 10 minutes. I trigger the following HTTP call when releasing an update to clear the edge cache locations:

curl -X POST "https://api.cloudflare.com/client/v4/zones/${$CLOUDFLARE_ZONE_ID}/purge_cache" \
     -H "X-Auth-Email: ${$CLOUDFLARE_EMAIL}" \
     -H "X-Auth-Key: ${$CLOUDFLARE_API_KEY}" \
     -H "Content-Type: application/json" \
     --data '{"purge_everything":true}'

Still, if you refresh this page, the hit counter below will increase:

Visits: [VISITS_COUNT]

No JS involved. What kind of sorcery is this!?

Basic Cloudflare worker with KV store

To implement this feature I’ve used Cloudflare Worker Routes. They work similarly to Lambde@Edge, i.e., workers sitting in front of the cache can reprocess a response before sending it to the client. Currently, it natively supports Javascript, Typescript, Python, and Rust, but more languages are supported by Wasm (Web Assembly). In this tutorial we will use Rust.

Unfortunately, Cloudflare currently supports creating only Javascript workers directly from the UI. So to get started you’ll need to install Rust locally.

Next run these commands to install necessary dependencies:

rustup target add wasm32-unknown-unknown
cargo install cargo-generate

And initialize your project using a template:

cargo generate cloudflare/workers-rs

Select a Hello world template and name your project. It creates the following worker implementation:

src/lib.rs

use worker::*;

#[event(fetch)]
async fn fetch(
    _req: Request,
    _env: Env,
    _ctx: Context,
) -> Result<Response> {
    console_error_panic_hook::set_once();
    Response::ok("Hello World!")
}

You can test it by running:

npx wrangler dev

Now you can access it at http://localhost:8787

Cloudflare worker process running locally

Let’s make this example more interesting by adding a visits counter using KV storage:

src/lib.rs

use worker::*;

#[event(fetch)]
async fn fetch(req: Request, env: Env, ctx: Context) -> Result<Response> {
    let kv = env.kv("visits")?;
    let visits = kv.get("count").text().await?.unwrap_or("0".to_string());
    let visits = visits.parse::<i32>().unwrap_or(0) + 1;
    let _ = kv.put("count", visits.to_string())?.execute().await;

    Response::ok(format!("Visits: {}", visits))
}

This example reads number of visits from a key-value store and increments it on each request. If you test it in the browser, you’ll notice that the counter increments by 2 on each visit. It’s because of the GET /favicon.ico request which also triggers the hit.

Let’s now deploy our counter to production. You have to start by creating a production key-value store:

npx wrangler kv namespace create visits

This command will prompt you to login to your Cloudflare account. Don’t forget to add the output to your wrangler.toml file:

wrangler.toml

[[kv_namespaces]]
binding = "visits"
id = "XXX"

We’re now ready to go live by typing:

npx wrangler deploy

Running this command will output the URL to access your first deployed Cloudflare worker.

Enabling Cloudflare workers proxy

Let’s make our worker more useful by hooking it up as a proxy to a live website. This tutorial assumes that you have a website that uses Cloudflare DNS with proxy enabled i.e. cloud should be orange:

Now go to Workers Routes in your domain’s settings and click Add route. Now select a page for which you want to activate your worker. As for Request limit failure mode select Fail open so that your page will keep working even if you exhaust the worker’s quota. On a free plan you can use 100k requests/day and up to 10 million a month.

Once you enable it, your website will display a blank page with a counter we’ve configured previously. Here’s how you can enable the proxy mode:

use worker::*;

#[event(fetch)]
async fn fetch(req: Request, env: Env, ctx: Context) -> Result<Response> {
    let kv = env.kv("visits")?;
    let visits = kv.get("count").text().await?.unwrap_or("0".to_string());
    let visits = visits.parse::<i32>().unwrap_or(0) + 1;
    let _ = kv.put("count", visits.to_string())?.execute().await;

    let mut origin_response = if env.var("WORKER_ENV")?.to_string() == "production" {
        Fetch::Request(req).send().await?
    } else {
        Response::from_html("<p>Visits: [VISITS_COUNTER]</p>").unwrap()
    };

    let body = origin_response.text().await?;
    let body = body.replace("[VISITS_COUNTER]", &format!("{}", visits));

    let response = Response::from_html(body).unwrap();
    let mut response = response.with_headers(origin_response.headers().clone());
    response.headers_mut().delete("Last-Modified")?;

    Ok(response)
}

We removed the Last-Modified header to avoid blank page errors, as described here.

We use a WORKER_ENV config variable, to allow testing our worker locally. You’ll have to create .devs.vars with the following content:

WORKER_ENV="development"

and add:

[vars]
WORKER_ENV = "production"

to wrangler.toml.

With this configuration in place, locally, you can work with a mocked version of an origin page. On production, we use Fetch::Request(req).send().await? call to download our target website and reprocess its body.

Our modification is just replacing all the occurences of the [VISITS_COUNTER] string with the count fetched from the KV store.

All you have to do now is embed the placeholder anywhere in your origin page and deploy the updated worker. The hit counter is live, and you can feel like the web design is back to its former glory.

Visits: [VISITS_COUNT]

Security headers with Rust Cloudflare workers proxy

But if visit counters are not your thing, the workers proxy offers great flexibility in modifying any part of the HTTP response.

A practical application of this is adding security headers to systems that otherwise don’t support it. I’ve known companies whose primary landing page used a niche CMS, without an NGINX proxy. So adding custom headers was not possible without a general infrastructure overhaul.

You can easily overcome this limitation by implementing the following worker:

use worker::*;

#[event(fetch)]
async fn fetch(req: Request, env: Env, ctx: Context) -> Result<Response> {
    let origin_response = if env.var("WORKER_ENV")?.to_string() == "production" {
        Fetch::Request(req).send().await?
    } else {
        Response::from_html("mock").unwrap()
    };

    let mut headers = origin_response.headers().clone();
    headers.set(
        "strict-transport-security",
        "max-age=31536000; includeSubDomains; preload",
    )?;
    headers.set("x-frame-options", "SAMEORIGIN")?;
    headers.delete("Last-Modified")?;

    Ok(origin_response.with_headers(headers))
}

With similar worker, you should always be able to score an A+ on securityheaders.com.

Summary

Rust, Wasm, CDN, and edge cache workers… I wonder how devs implemented hit counters 20 years ago without all this advanced tech. Anyway, Cloudflare Workers is an interesting tool that is worth knowing. In some cases it could offer a simple solution to otherwise complex devops tasks.

Ruby and Rust developer specializing in backend performance - Paweł Urbanek

Rustikon Talk - Rust Performance Profiling using hotpath-rs

Links and resources

Lessons Learned Building High-Performance Rust Profiler

hotpath profiler 101

Live profiling TUI

Profiling the profiler

Fixing the MPSC channel bottleneck

Alloc metrics CPU cache busting

Async functions memory tracking

CPU profiling integration

Interpreting raw CPU traces

Exclusive vs inclusive CPU attribution

CPU delta vs. samples count

inline compiler hints

Summary

How to Automate MEV Analysis on EVM Chains using OpenClaw MCP

Initial setup and RPC requirements

Configuring an MCP server with mevlog-rs

Configuring a remote MCP endpoint for better RPC performance

How to query EVM blockchains with the OpenClaw LLM agent

Summary

Using LLMs and MCP to Debug PostgreSQL Performance in Rails

How to optimize PostgreSQL performance with AI?

Debugging context provided by the pg-extras gem

Automating pg-extras Performance Debugging with LLMs

Legal and security concerns of using LLMs with production data

Enabling optional EXPLAIN ANALYZE mode

How much does it cost?

Summary

My puts Debugging Workflow in Rails Apps

Why and how to puts debug Ruby code?

Introducing dbg-rb - a simple Ruby puts debugging helper

How to use dbg-rb in Rails without adding it to the Gemfile?

Summary

How to Extract long-tail MEV Profit from Uniswap

How to discover and profit from imbalanced ERC20 pools?

One exploit a day

How to find less competitive MEV strategies?

Simulating MEV profit opportunity with Foundry

Step three is profit

Summary

How to Discover long-tail MEV Strategies using Revm

How to monitor EVM chains for MEV opportunities?

Revm-powered EVM tracing

How to analyze MEV sandwich attacks?

SQLite and ENS integration

Optimal EVM tracing strategies with Revm and RPC

Future plans

Summary

Lessons Learned Migrating my SAAS to Rails 8

Kamal vs Dokku

Kamal registry-based deployments

Goodbye Redis

Measuring non-memory caching performance

Using a custom compressor for Rails cache

SQLite vs PostgreSQL

Summary

How to Send deadbeef Transactions with Alloy

Vanity txs

How to brute-force ETH tx prefix?

Benchmarking tx prefix generation

Summary

How to use Cloudflare Workers proxy with Rust

Static blog with CF edge caching

Basic Cloudflare worker with KV store

Enabling Cloudflare workers proxy

Security headers with Rust Cloudflare workers proxy

Summary

`inline` compiler hints

Introducing `dbg-rb` - a simple Ruby puts debugging helper