What type of performance should I expect from my Threadripper? (9093 CPU errors)

[snowplaysmusic]

Hello. I have posted elsewhere about some of the challenges I am having with my 3990x Threadripper. I am about to post system information, tweak history, BIOS settings, and a Sandra report, but first I wanted to ask the general question about performance to set my expectations.

In a session with nothing but plugin-free empty audio tracks, how many tracks should a top-end system be able to simultaneously record at 32 buffer, 48 khz without CPU errors or other fall outs? Are we talking 4 tracks, 20, 250?

I can record 8 tracks on average between 4-11 minutes before I get a 9093. This is repeatable.

Would this be expected for a top end system of the past 2-3 years, or am I vastly underperforming? Is this a standard limitation of HD Native systems to run them at such low buffer/latency, or not even close? Once I get a general expectation of how systems are behaving, if I am underperforming I will go ahead and start posting some more information. This is important to me because I am trying to do low latency cue mixes and tracking drums/live bands.

[Darryl Ramm]

Your system is operating way way way below expectations for any decent computer which several of us have pointed out to you here recently.

But be careful of comparing just tracking audio at super low IO buffers as you are really in the noise there, and results for different systems will vary greatly based on who knows what. So it's maybe also worth comparing 64 and 128 sample performance, and with some plugin or background playback, load on the sessions.

And what sample rate? What interface? What driver? At these limits the interface and driver becomes critical.

As an example of a quick test as you describe, on my 15" MacBook Pro 2019 with 8 core i9 I'm recording 100 tracks mono @ 44.1kHz, macOS Activity Monitor reports Pro Tools is using ~125% CPU (i.e.; somewhere between 1-2 of 16 virtual cores... this is never going to be a CPU constrained problem even on middle of the road systems). 32 sample IO buffer, ignore errors unchecked, no funky routing, all track output directly to a physical output. Optimize small buffer performance and enable Intel Turbo processing options are checked (apply only to Macs). 2GB disk cache. Pro Tools 2022.7. RME Fireface UFX+ Interface firmware v54, driver v3.29. macOS Monterey 12.5.1. And the MBP is driving both its internal display and a 5K external display, it's running a couple of tabs of Google Chrome (including where I'm typing this reply) and macOS Activity Monitor and some RME config utilities. I have no interest in simultaneously tracking beyond a small number of tracks, not going to push this to it's breaking point. But let me check... been recording 100 tracks for over 26 minutes.

But it's not just that you have a AMD Threadripper system that is interesting, but IIRC you have the largest Threadripper processor config possible with 64 core/128 threads. You've got that water cooled which is great. I would be surprised if Avid tested close to that config. And I'd be surprised if many users are running Pro Tools on such a large Threadripper based system--because of the cost of the processors at that size and what I expect to be a widely held (and likely correct) expectation of diminishing scaling and cost/benefit returns of such an extreme #core system.

Do you have an open case# with Avid support for this problem. If not you should open one. And as a part of that ask Avid what Threadripper systems they have tested on... what's the largest? Or if not Threadripper what's the largest number of cores on an Intel system? The largest system I've seen mentioned for Pro Tools is the HP Z6 qualified system with dual 16 core processors, so 32 core/64 threads.

Avid and/or Lenovo did some work qualifying P620 systems for Media Composter, even though Pro Tools is mentioned I wonder if it was really looked at. See https://resources.avid.com/supportfi...%20Rev%20C.pdf

And I know your system is not a P620, but I expect asking about Threadrippers will likely be mostly a discussion about P620s. Given how much all the early Threadripper deployments were P620s and Lenovo just dominates the Threadripper workstation market.

I like the AMD Threadripper and it's EPYC cousin a lot, have run some machine learning and other code on Lenovo P620s and almost brought a P620 for myself (not for Pro Tools). It would have been interesting to see Avid qualify some of the Lenovo P620 based Threadripper systems. Would have made a nice change from endless HP boxes.

---

It would be great if you post those notes about optimization, Sandra report etc. otherwise you'll end up with folks going over stuff already discussed in your other threads. But it is worth having the best windows PC folks here go over what you have done, even if they likely have no Threadripper experience. Post them that here, please stop creating new threads about this.

Been recording 100 tracks for ... over 60 minutes... (ha and it just took a AAE 6101 error when I tried to take a screen shot of all the tracks).

[snowplaysmusic]

First I sought to isolate-away the HD Native PCIe card and DigiLink cable. Growing up under a test engineer father, these things do go wrong. I bought an HDX card. The results: With HDX hybrid engine I was able to record 196 DSP-enabled, extreme low latency tracks with my overall CPU usage around 5% or less and my HDX card less then half utilized. I bet I could push 400 with no flicker. However, this only works if I have the buffer size set to 1024 samples. This makes sense, as the HDX buffer size is "fixed" and the GUI buffer setting is only relevant to native processing. Setting the buffer to 32 (or 64 samples), I still could not record 8 tracks (even pure DSP-enabled tracks) without CPU 9093 and 9171 errors. In fact, I am spiking at 128 samples as well. HDX will solve my core studio requirements for tracking and headphone mixes as extreme low latency. So here I am pleased, quite impressed with HDX, and could call it a day. But relating to the original problem statement, the results of this test show that there is not a problem with the HD Native PCIe card nor DigiLink cable, nor an issue with running the mix engine on the host CPU.

Next I sought to isolate away the Threadripper. I bought a 12900K pre-built. I have both machines running in-tandem. I migrated over my HD Native PCIe and DigiLink cable. I optimized with ThisIsWin11, disabled all startup items and services. LatencyMon is stable, reporting a remarkable 1 microsecond DPC. For context, my Threadripper reports a still extremely-good 4 microsecond DPC. I have read people who consider 100-200 microseconds good for pro audio. My machines are performing stellar.

The results? Shockingly, the 12900K spikes out almost instantly at 32 buffer with 8 tracks. Process Lasso shows an extremely lean, optimized set of services, with only roughly 3 services consuming any CPU time, and no real spiking.

The only system I have not isolated away would be my interface, the HD IO. I doubt I have a power problem as it's a well-to-do house built in 2010. I can't think of anything else besides cosmic X-Rays targeting my house from space.

[snowplaysmusic]

The tests are at 44.8khz, 32 buffer, on a 2018-era HD IO 8x8 (only analog, no digital card [I was a broke college kid who spent all his loan money on analog])

There may be 5 or less running the 3990x for Pro Tools I dropped $4k on this processor on release day. Just a day before I bought the 12900K (yesterday), I reasoned myself into a conclusion that the 3990x is just not a good audio processor. The rationale was that the clock speed (or more precisely IPC) is the limiting factor of low latency tracking, and that trading away ~1GHz clock speed to gain the extra 48 cores (compared to other systems), as well as stretching the processing out through long connections through the infinity fabric (and other architectural concerns with a 64 core/128 thread chip that I couldn't imagine) was the key problem behind my CPU spiking. But the 12900K is a respectable machine, and isolates away all the weirdness of the Threadripper platform. Additionally, it should be able to spar toe-to-toe with the latest M1 macs. That it's spiking right away, after software optimization and the most minimal hardware configuration, blows my mind. This isn't to diss Avid, and if not for your (and other reports) of your ability to track multiple tracks at 32 buffer, I would have just made the conclusion that native platforms do not have the technical capabilities of solving this problem, and that's what HDX is for.

I never brought it up with Avid. I read their support was so bad. You're probably right, I should, no harm will come from it.

[snowplaysmusic]

ASUS ROG Zenith II Extreme Alpha
AMD 3990X
4x G.Skill DDR4 3200 quad-channel RAM (128gb total)
4x Samsung 980 Pro NVME M.2
1x Samsung 970 Evo+ NVME M.2
NVIDIA GeForce RTX 2070
Corsair AX1600i 1600 watt PSU
Premium EKWB CPU block, three 480mm radiators, x11 140mm fans, low temps

Software Optimizations
Secure erase Samsung 980 Pro with Heatsink in M.2_1 slot
Windows 11 (21H2) offline installation (disable all privacy and options)
NVIDIA Studio Driver (disable all options but core driver)
- 1080p resolution at 100% scale
- Max Performance in NVIDIA settings
Upgraded onboard audio drivers
Disable real-time protection
Disable defender firewall
Turn all unnecessary windows features off
Ryzen High Performance power plan (also tried Ryzen Balanced, Windows Ultimate Performance, and Bitsum Highest Performance)
Advanced system settings
-> Turn off hard disk: never
-> Desktop slideshow: paused
-> Wireless adapter: Maximum power saving
-> USB selective suspend: disabled
-> Tried both background vs foreground processes
System Protection Off
Uncheck allow files on this drive to have contents indexed
Updated AMD Chipset Driver (from AMD website)
Latest Grace m905 Driver
Latest Pro Tools HD driver & Pro Tools
CPU-Z, HwInfo64, LatencyMon, Process Lasso, Samsung Magician for monitoring (tried with ProcessLasso and without, including CPU affinities and priorities)
SSD firmwares upgraded
Latest Marvell LAN Driver
Latest Windows Update
Disable everything unused in device manager (Audio inputs and outputs)
Optimized with LoveWindowsAgain
Optimized with ThisIsWin11
msconfig -> Disabled all non-microsoft services (AsusUpdateCheck, NVIDIA Display Container LS)
Task Manager -> Disabled all startup applications (OneDrive, Windows Security notification, Avid Audio MME Binder startup, Windows Audio, Downloaded Maps Manager, Windows Search, Radio Management, Touch Keyboard)
services.msc -> Disabled all unnecessary services (Print spooler, bluetooth, telemetry, network discoverability, etc.)
Uninstall -> Avid Cloud Client Services, Avid Codex LE, Avid Link, Bonjour

BIOS optimizations (tried hundreds of permutations)
Disabled SMT hyperthreading
Disabled C States
Disabled Turbo Boost (runs better with it on)
Disabled SATA controllers, HD audio, Bluetooth, Wifi, Networking
No overclocking
Latest BIOS

I do believe these superset is the greatest audio optimization for PCs known to man. I could write a book on this and sell it. Again, I'm down to 4 microsecond DPC latency on the Threadripper and 1 microsecond DPC latency on the 12900K with no spiking or interrupting.

I'm not any closer to solving this issue. I still do not know what to say. It's either that 1) HD Native systems can run low latency tracking and I have extraordinary bad luck 2) HD Native systems cannot run low latency tracking and rare people have extraordinary good luck 3) Cosmic X-Rays are shooting down onto my house 4) It's the HD IO (the only component I did not isolate). The MTRX Studios are on backorder. I'm taking a vacation to the beach. I've worked so hard on this I am about to have a heart attack.

[Darryl Ramm]

Lots of good info here, but where is the Sandra report?

I assume with HDX only, non-hybrid you can max out the HDX card?

HDX does not itself have a "buffer size" so no it's not "fixed".

Are the exact DIMM part numbers installed on the vendor qualified list for this Mobo?

The BIOS DIMM/memory settings are all correct? slightest screw up in memory can cause all sorts of crap.

You have these machines loaded up with *stuff*: Process Lasso, LoveWindowsAgain, ThisIsWin11 I doubt many folks here use any of those utilities and have any idea what they do or know exactly what you have set any of this to do. At best these are likely unneeded, at worse, who knows.

There are manual instructions for affinity settings, which are critical for some windows systems. Well described in the Avid Knowledge base here: https://avid.secure.force.com/pkb/ar...finity-Windows, have you followed those?

Did you clone the Threadripper system to the new one or do a full new install on it? Either way, both of your impressive powerful systems that have ended up with similar problems running native. The natural common suspect here is what exactly you have been doing setting them up. So that is where to focus if you want to keep working on this.

On either of the systems (you have one to always dedicate to troubleshooting now ) I would be doing a clean Windows install with none of these third party utilities, no other apps, no AV software, as little motherboard OEM crapware as absolutely needed, I would not let Windows mess with drivers, install manually the bare minimum GPU driver and just install Pro Tools and test, doing optimizations by hand.

I would be physically disconnecting as much stuff as possible when getting started. Pull out all but one SSD you are installing Windows on, physically unplug everything else you can. No m905, no USB drivers, no plugin installs, no other apps, none of this extra stuff until you have a tests with Pro Tools that is not running like utter crap when native.

For some of the BIOS settings you can guess what will be better on such a scaled out system and just leave stuff set that way, e.g. disable hyper threading and leave it off. Do all other recommended settings and leave them that way. That applies to all optimizations, you are not trying to test each one to see what difference it makes, with such a sick system just plow though making sure everything is done, and leave them done while you move on. Like you say you disable BT and WiFi.. but I hope that means you disable them and leave them off while you keep going with other stuff. Yes any modern system should be able to run with them on, but a sick system... who knows, just crush everything by using the safest/fully optimized system.

I would love to see you get that Threadripper box working well, such a great beast and great AMD technology.

[EGS]

Originally Posted by snowplaysmusic

... low latency cue mixes and tracking drums/live bands.

Does it work at buffer 64? I have no Threadripper experience, but FWIW my i9-11900K rig can record 16 tracks at once with plugs for hours no errors @ at buffer 64. I'm assuming you've done:
1) all AVID optimizations
2) all AVID suggested troubleshooting steps
3) the AVID suggested affinity switch

https://avid.secure.force.com/pkb/ar...ssage/en336705

It seems you've done a lot already, but for now, check all 3 above again. Skip any, and it just might come back to bite ya...

[snowplaysmusic]

Originally Posted by Darryl Ramm

Lots of good info here, but where is the Sandra report?

I believe Sandra can't do the insane number of cores / threads and is why it crashes during benchmarking. I wish I could provide one for us

Originally Posted by Darryl Ramm

I assume with HDX only, non-hybrid you can max out the HDX card? HDX does not itself have a "buffer size" so no it's not "fixed".

I am going to try to max it out today. But it appears this issue is solely about the native buffer. I am extremely confident I could record 400+ DSP tracks without a hiccup (since I can do it at 1024 buffer). But I will verify it today.

Originally Posted by Darryl Ramm

Are the exact DIMM part numbers installed on the vendor qualified list for this Mobo? The BIOS DIMM/memory settings are all correct? slightest screw up in memory can cause all sorts of crap.

Fantastic point, and something I overlooked. The RAM is not on the QVL. In fact, the minimum memory listed is 3600MHz and mine is 3200MHz. It's also quad channel memory, which truly deserves faster rated sticks then specs (not overclocked, just that it's quad channel and a heavy burden on the board and controller). I am going to pick up new RAM and see.

Originally Posted by Darryl Ramm

You have these machines loaded up with *stuff*: Process Lasso, LoveWindowsAgain, ThisIsWin11 I doubt many folks here use any of those utilities and have any idea what they do or know exactly what you have set any of this to do. At best these are likely unneeded, at worse, who knows.

I was anticipating this point. I think it's explained that I arrived at these solutions over two years of troubleshooting. The optimization logs I posted above have been done in step-by-step fashion with testing in-between. I am have boiling up the most-certain steps towards the top and less the certain steps towards the bottom. I am still prioritizing and improving the steps. I am taking SDD images with Macrium with detailed logs of BIOS/Software changes and creating a chronological history. The items in my list I feel confident with because of the information ProcessLasso demonstrates. Also, the *stuff* as you say is almost all the reduction or elimination of superfluous things, or programs that have no registered services and do not start-up unless manually invoked. When I start Pro Tools, I'm running less services and programs then any configuration I've ever seen. I don't imagine the subtraction of things to be a problem, but your point does have some good validity. I am happy to try any permutations of the above. I mostly have already, but of course with so many tweaks it becomes a 2 to the power of 2 problem and it explodes combinatorially. But I did try to do the "proven" or most expected fixes first in chronological order with testing before moving on to more reductions.

Originally Posted by Darryl Ramm

There are manual instructions for affinity settings, which are critical for some windows systems. Well described in the Avid Knowledge base here: https://avid.secure.force.com/pkb/ar...finity-Windows, have you followed those?

Yes, ProcessLasso is a wonderful tool for affinity (as well as priority). I have tried all permutations I can think of. For example, Pro Tools on cores 1-63 (giving 0 to OS processes), giving Pro Tools odd number cores (with SMT enabled), giving Pro Tools 36-64 (roughly half the chip), and 24-48 (center cores which get the most load).

Originally Posted by Darryl Ramm

Did you clone the Threadripper system to the new one or do a full new install on it? Either way, both of your impressive powerful systems that have ended up with similar problems running native. The natural common suspect here is what exactly you have been doing setting them up. So that is where to focus if you want to keep working on this.

The new system was a fresh Win11 install with testing. Testing did not do well, then I proceeded with the next basic steps which were disabling startup programs and turning off hyperthreading and C States. Testing still did not do well, then I proceeded to the more nerdy tweaks like disabling services and using Process. Testing still did not do well, then I went to affinitys, ThisIsWin11 stripping. At that point I had to stop testing and return the system.

Originally Posted by Darryl Ramm

On either of the systems (you have one to always dedicate to troubleshooting now ) I would be doing a clean Windows install with none of these third party utilities, no other apps, no AV software, as little motherboard OEM crapware as absolutely needed, I would not let Windows mess with drivers, install manually the bare minimum GPU driver and just install Pro Tools and test, doing optimizations by hand.

Yes yes, this is how I do it everytime, with the exception that Windows update snuck in (behind my back!) and gave me some drivers.

Originally Posted by Darryl Ramm

I would be physically disconnecting as much stuff as possible when getting started. Pull out all but one SSD you are installing Windows on, physically unplug everything else you can. No m905, no USB drivers, no plugin installs, no other apps, none of this extra stuff until you have a tests with Pro Tools that is not running like utter crap when native.

We are on the same page. All my testing procedures have been starting with 1 SSD, GPU, no USB, no m905, no plugins, bare minimum hardware configuration, extra PCIe ripped out, etc. I'm down to 4 sticks at 128gb (that's the lowest I should go with quad channel memory) but I actually have 8 sticks and 256gb total. My point is my methodology is stripping away EVERYTHING and starting with nothing. I secure erase, connect the HD IO, and go from the bare basics and proceed to the more uncertain tasks.

Originally Posted by Darryl Ramm

For some of the BIOS settings you can guess what will be better on such a scaled out system and just leave stuff set that way, e.g. disable hyper threading and leave it off. Do all other recommended settings and leave them that way. That applies to all optimizations, you are not trying to test each one to see what difference it makes, with such a sick system just plow though making sure everything is done, and leave them done while you move on. Like you say you disable BT and WiFi.. but I hope that means you disable them and leave them off while you keep going with other stuff. Yes any modern system should be able to run with them on, but a sick system... who knows, just crush everything by using the safest/fully optimized system.

Yes, indeed. The proven "axioms" so far are SMT disabled, all unnecessary onboard devices disabled (network, LAN, Bluetooth, WiFi, SATA), C states disabled, turbo boost enabled (this just should not be turned off on Threadripper), all software services and startup items disabled. I have allowed Windows 11 updates into this early process, which does seem to add a few drivers.

Originally Posted by Darryl Ramm

I would love to see you get that Threadripper box working well, such a great beast and great AMD technology.

Thanks Darryl. I really appreciate working it out together. It's been fun. My hope is that the community sees our collaboration and can learn something. I have a solid background in software engineering (so when pro tools crashes with the rare std::unique_ptr<> line I can interpret what it means) and I understand much about the software hardware interface. I have learned so much from this journey. In fact, while my system may not be performing great at super low latency, I am coming to a realization. This beast of a machine is not a low latency machine, it is a mixing/post production powerhouse. At high samples I can record eight 1080p video streams to disk, stream RMTP to Twitch/wherever, drive four TVs for video monitoring, record hundreds of tracks, have 100 gig+ of virtual instruments loaded, all at the same time with 75% of my cores ready for more work. Even 2 years later, it is ridiculous performance. But I don't think this is a low-latency tracking machine. I am really incredibly impressed with HDX. It's almost like I have the best of both worlds... for mixing max out the buffers and do all the video and plugins and multi-processing I want, and when I have latency requirements, run on the HDX card.

By the way, I just replaced the graphics card with an RTX 550 and minimal AMD drivers. I put the HDX card in PCIE_16_1 and GPU in PCIE_16_3. I never considered putting anything but the GPU in the first slot, but both slots run x16, and I found a post of a hardware builder on DUC who said he solved many CPU issues by changing slot order. The results? Still spiking.

I think next is swapping out the RAM. I'm probably under clocked for quad channel memory, and even 2 years ago this kit could never reach the expected timings that it "should" have been able to. It does run XMP 3200 but admittedly at pretty weak timings, and like mentioned earlier it's not on the QVL.

Originally Posted by Darryl Ramm

I would love to see you get that Threadripper box working well, such a great beast and great AMD technology.

Thanks Darryl. I really appreciate working it out together. It's been fun. My hope is that the community sees our collaboration and can learn something. I have a solid background in software engineering (so when pro tools crashes with the rare std::unique_ptr<> line I can interpret what it means) and I understand much about the software hardware interface. I have learned so much from this journey. In fact, while my system may not be performing great at super low latency, I am coming to a realization. This beast of a machine is not a low latency machine, it is a mixing/post production powerhouse. At high samples I can record eight 1080p video streams to disk, stream RMTP to Twitch/wherever, drive four TVs for video monitoring, record hundreds of tracks in Pro Tools, have 100 gig+ of virtual instruments loaded, all at the same time with 75% of my cores ready for more work. The trade-off is latency. Even 2 years later, it has ridiculous performance. But I don't think this is a low-latency tracking machine. I am really incredibly impressed with HDX. It's almost like I have the best of both worlds... for mixing max out the buffers and do all the video and plugins and multi-processing I want, and when I have latency requirements, run on the HDX card. The only thing I can't do is virtual instruments at low latency. But I have an XK-5 and Leslie 3300 so I tell myself just keep the HDX card, count your blessings, and play your organ :)

By the way, I just replaced the graphics card with an RTX 550 and minimal AMD drivers. I put the HDX card in PCIE_16_1 and GPU in PCIE_16_3. I never considered putting anything but the GPU in the first slot, but both slots run x16, and I found a post of a hardware builder on DUC who said he solved many CPU issues by changing slot order. The results? Still spiking.

I think next is swapping out the RAM. I'm probably under clocked for quad channel memory, and I remember even 2 years ago this kit could never reach the expected timings that it "should" have been able to. It does run XMP 3200 but admittedly at pretty weak timings, and like mentioned earlier it's not on the QVL, and slower then the minimum rating RAM. Amazing oversight on my part.

Even if I can't get the 9093 issues away, I still have an incredible systems solution with the combination of Threadripper X-Ray detector system and HDX. It does everything I need. I could pack up this journey, move on and not be effected. And that's great because the best thing a technology system can be is INVISIBLE to you. HDX is a huge win for me. But I am continuing this journey for the good of the community and because I am intellectually curious.

[snowplaysmusic]

Originally Posted by EGS

Does it work at buffer 64? I have no Threadripper experience, but FWIW my i9-11900K rig can record 16 tracks at once with plugs for hours no errors @ at buffer 64. I'm assuming you've done:
1) all AVID optimizations
2) all AVID suggested troubleshooting steps
3) the AVID suggested affinity switch

https://avid.secure.force.com/pkb/ar...ssage/en336705

It seems you've done a lot already, but for now, check all 3 above again. Skip any, and it just might come back to bite ya...

Good point, I really should take a step back (one more time) and be meticulous for the proven things. These optimizations are generally the first things I've been doing. Some items I believe AVID's recommendations are not right anymore. For example, setting system settings for 'background processes' I don't think is the best. And Turbo Boost just should be turned off for Threadrippers. But I only arrived at these conclusions with actual testing. Thank you for your data point, it is helpful and good to have another comparison. Once I get back from vacation, I am going to do a fresh disk image one more time with a fresh kit of RAM and be a little more detailed and explicit on which steps I take and in which order. The best optimizations should happen first, and the winners should be kept, before proceeding on to the more uncertain tweaks. The name of the game is real data, precise record keeping, proper order of events, and minimizing changing variables when progressing to the next round. It's like a little science experience, which is a fun and mind-expanding journey.

[albee1952]

I agree that your machine is not living up to its promise, but I also don't know what's up with the i9 system either. As a point of reference(and to maybe dispel doubts about the HDN card), my mix machine is a fairly old i7 3930K, X79 Asus Sabertooth, with 32GB RAM and nVidia Quadro K2000 graphics and I run 50-60 track sessions with dozens of plugins at a 64 buffer. My previous recording/mixing rig was a used HP z820 12 core(@3.5GHz) with 64GB of RAM and a GeForce 750Ti and I tracked full bands(64 buffer) 5 days a week for years. Again, this was with dozens of plugins and providing 6 stereo headphone feeds via sends on every track. Both of these rigs run(ran) HDN cards(current rig runs a single 192 and HP rig ran pairs of HD IO and 192 IO), Windows 10 Pro and always the latest version of PT and I have not seen a 9093 error in at least 5 years

Part of troubleshooting needs to be pulling all 3rd party plugins out of the Plugins folder>test>return small batches of plugins(as in 1 brand at a time) and test again. Maybe you can pin this on a plugin(which can mess up, even if the plugin is not actually used in a session).