# My Research Journey into Rust & Performance: Solving the 1BRC Challenge ⚡️

A little over a year ago, I got curious about the **1 Billion Row Challenge (1BRC)**. It seemed like the perfect playground to test Rust’s performance chops — 1 billion weather station measurements, aggregate per-city statistics (min, max, average), and do it as fast as possible.

At that time, I went down a rabbit hole of **Rust performance research**, experimenting with naïve approaches, multithreading, and low-level optimizations. I never wrote about it back then, but looking back, the lessons are worth sharing. So here’s my journey — from **12 minutes → 2 mins → 10 seconds**.

---

## Stage 1: The Naïve Rust Approach — 12 Minutes ⏳

I began with a straightforward solution:

* Load the file into a string.
    
* Split by newline.
    
* Parse each line into `city;temperature`.
    
* Aggregate results in a `HashMap<String, CityStats>`.
    

It was **idiomatic Rust**, safe, and simple. But it took **12 minutes** to finish.

This stage gave me a baseline, but it was clear that high-level string parsing was eating performance alive.

---

## Stage 2: Embracing Concurrency — 15 Seconds 🚀

My next line of research was **parallelism**. Rust provides great abstractions like `std::thread::scope` and `Arc<Mutex<T>>`, so I divided the file into **thread-safe chunks** aligned on newline boundaries. Each thread processed its own slice of the file and then merged results into a global `HashMap`.

The speedup was dramatic — **down to ~2 mins**.

This was my first “wow” moment: Rust’s **fearless concurrency** makes scaling across CPU cores approachable and safe. But something was still bothering me — parsing overhead.

---

## Stage 3: Researching Parsing Costs → Working with Bytes — 10 Seconds ⚡️

I dug deeper into how Rust handles strings and UTF-8. My research led me to an important insight:

> **Strings are expensive. Bytes are cheap.**

Every conversion to `String` or `&str` was adding overhead. So I restructured my code to work directly on **raw** `u8` arrays. Instead of treating the file as text, I processed **byte slices** and converted only when strictly necessary.

This optimization cut execution time almost in half — **from 2 mins to ~10s**.

At this point, profiling showed something surprising:

* **~4s** = actual computation.
    
* **~6s** = just loading data from the SSD.
    

That meant I had reached the **I/O limit of my hardware**. Any further improvement would require tricks like memory-mapped files (`mmap`), SIMD parsing, or asynchronous I/O.

---

## Lessons Learned 📚

This wasn’t just about solving a coding challenge — it was a **research journey** into Rust’s performance model.

1. **Naïve is necessary.** My 12-min baseline gave me something to measure against.
    
2. **Concurrency matters, but parsing dominates.** Threads gave me my first big win, but eliminating string parsing was the real breakthrough.
    
3. **I/O is king.** Once your code is fast enough, the bottleneck shifts from CPU to hardware.
    
4. **Rust shines in performance-critical paths.** Working with raw bytes in a safe way is exactly where Rust feels both low-level and empowering.
    

---

## Code Snapshot: Processing Data with Bytes

Here’s the core of my final approach:

```rust
fn process_data(data: &[u8]) -> HashMap<String, CityStats> {
    let mut map: HashMap<String, CityStats> = HashMap::new();

    for segment in data.split(|&byte| byte == b'\n') {
        let mut parts = std::str::from_utf8(segment).unwrap().split(';');

        if let (Some(city), Some(value)) = (parts.next(), parts.next()) {
            let val = value.parse::<f32>().unwrap();
            match map.entry(city.to_string()) {
                Entry::Occupied(mut e) => {
                    let s = e.get_mut();
                    s.count += 1.0;
                    s.sum += val;
                    s.min = s.min.min(val);
                    s.max = s.max.max(val);
                }
                Entry::Vacant(e) => {
                    e.insert(CityStats { min: val, max: val, count: 1.0, sum: val });
                }
            }
        }
    }

    map
}
```

---

## Closing Thoughts 💡

This project was less about “solving 1BRC” and more about **understanding Rust at the performance frontier**.

I started with high-level Rust (strings, safe iteration) and ended up optimizing down to raw bytes. Along the way, I learned how **multithreading, memory access patterns, and I/O limits** interact in real-world workloads.

Right now, my solution runs in **10 seconds**, where **6 seconds are I/O bound**. That means the core algorithm is blazing fast — and any further speedup requires going beyond CPU optimizations into **system-level tricks**.

This experience has convinced me: **Rust isn’t just about safety. It’s about giving you the tools to write code that’s as fast as your hardware will allow.**