Hashrate variance on Intel Arrow Lake, improved by background migrating thread
On my new Intel 265K (8 P cores, 12 E cores, no SMT), I'm seeing results like this with repeated --bench=1M runs:

    117.992 seconds (8475.2 h/s)
    115.892 seconds (8628.7 h/s)
    114.032 seconds (8769.5 h/s)
    113.521 seconds (8808.9 h/s)
    112.626 seconds (8878.9 h/s)
    112.520 seconds (8887.3 h/s)
    111.918 seconds (8935.1 h/s)
    111.753 seconds (8948.3 h/s)
    111.161 seconds (8996.0 h/s)
    110.625 seconds (9039.5 h/s)

(Sorted, so it's easy to eyeball the distribution. There was no decrease-with-time evidence of heat soak before the sort.)

The machine is *extremely* quiescent. According to top -d60, `xmrig` is getting 1995% CPU out of a possible 2000%, and the balance is made up by time spent servicing interrupts.

The topology is that the P-cores have 3MiB L2, and the E-cores are in clusters of 4 with 4 MiB of shared L2, and then the whole thing has 30 MiB of L3. The best performance, so far, has been with all cores loaded up. Idling 2 threads in each E-core cluster does not seem to help.

I would like to understand this so that I can 1) know if I have a hardware problem, and 2) hopefully fix it so that I can get the high end of the distribution all the time, or at least not have to repeat the benchmark so many times to detect small perf differences between BIOS versions/settings.

----

More investigation, after the above was written:

(For scripts and detailed results, see [yump/xmrig-turbostat-anomaly](https://github.com/yump/xmrig-turbostat-anomaly). If you have an Arrow Lake, Lunar Lake, or any other Intel hyrbid CPU, please check and see if `ecore-tickler.py` improves performance on your machine.)

Found a clue on the high variance of xmrig results: turbostat running in the background at 5 Hz adds ~800 kH/s, even when all it's doing is `--show usec`.

10 hz seems to work _slightly_ better than 5 hz, but it's a negligible difference.

Doesn't work if it's not running on E-cores. Doesn't work if's running on only 1 E-core. Minimum reproducer seems to be:

    turbostat --quiet --show usec --interval 0.1 --cpu 8-19

Recorded a pair of scheduler traces of xmrig running with (good.trace.dat) and without (bad.trace.dat) the background turbostat at 20 Hz. The command to do that is:

    sudo trace-cmd record -e sched -- sleep 0.5

Analyzed as follows:

1. Open in kernelshark.
2. Filter to one cluster of E-cores with Plots -> CPU.
3. Filter out `sched_stat_runtime` events by unchecking them in Filter -> Events.
4. Zoom in on one group of pips.
5. Double click the first, to attach marker A to that one and scroll the table to the proper timestamp.
6. Swtich to marker B, and arrow-key through the table to see what's going on.

Turbostat's behavior is that it wakes on the last E-core at the beginning of the sampling interval, then migrates rapidly from e-core to e-core in sequence, starting with the first (8), presumably by setting its affinity, then goes back to sleep.

To eliminate variables, I implemented `ecore-tickler.py`, found in this same directory, which rotates its own affinity in a similar fashion, but does absolutely nothing else.

Initally, it was crashing out due to negative sleep lengths caused by wakeup latency (really, linux?). Wrapping with `sudo chrt 1` worked as a bandaid fix, but I fixed it to run without that.

Although, the tickling seems to be _very_ slightly more effective with chrt, just based on eyeballing the hashrate while mining monero.

Effect of tickling at 5 hz, perfgov:

`xmrig --bench=1M` on e-cores only, `MSR_PREFETCH_CONTROL = 0xa5`, which is 0x25 + the bit the disables the Array of Pointers prefetcher, which is locked as of BIOS 2.22.AS05. Single run, avg/peak:

* Untickled: 4992 / 6444
* Tickled : 6444 / 6457

Conducted further tests of various tickling parameters, with `xmrig --bench=1M` on all cores, `MSR_PREFETCH_CONTROL = 0x80`, which is default as of BIOS 2.22.AS05. Detailed results in `test_tickler_params_result.txt`. Summary of 5 runs:

| prio        | timing  | interval | median kH/s | peak kH/s |
| ----------- | ------- | -------- | ----------- | --------- |
| no tickling | n/a     | n/a      | 8813.7      | 8841.6    |
| realtime    | sync    | 0.2      | 9720.3      | 9854.6    |
| realtime    | sync    | 0.1      | 9882.0      | 9954.7    |
| realtime    | sync    | 0.05     | 9810.8      | 9970.5    |
| realtime    | stagger | 0.2      | 9829.4      | 9861.1    |
| realtime    | stagger | 0.1      | 9921.2      | 9999.9    |
| realtime    | stagger | 0.05     | 9894.1      | 9876.2    |
| sched_other | stagger | 0.1      | 9670.1      | 9876.2    |

"Best" appears to be realtime priority stagger at 10 Hz.
