CPU Spikes, 12.5, and processor affinity... Work-Around?

[59.9hz]

Windows 8.1; 12.5 vanilla; 12core i7

I've been experimenting with various settings of processor affinity, scheduling priority and different combinations of programs running while watching the System Usage window. I can repeat going from 100% CPU spikes with audio drop-outs to 0% CPU usage just by changing processor affinity under controlled circumstances.

I'm not through testing yet, but I was wondering if anyone knows anything about setting processor usage. I recall reading in the past about there being some kind of processor throttle in the Hardware Engine settings, but I think that might have been for HD. Is this setting still around? Also, I ran across a few people who suggested "not using" 1 processor core, but I can't see how to do that in Pro Tools.

Also, I was wondering if anyone is actively testing for CPU spikes... I'd like to compare notes.

[mkosacek]

I think the ability to use a # of CPUs was taken out of PT awhile back, at least I don't recall seeing it since the LE days. You might experiment with using hyperthreading or not (BIOS setting). I'm still on the fence whether this helps or not. With PT9 and earlier I had to disable HT but I'm running 12.4 today and it seems to work OK either way so far.

[TAMBOR]

Originally Posted by 59.9hz

Windows 8.1; 12.5 vanilla; 12core i7

I've been experimenting with various settings of processor affinity, scheduling priority and different combinations of programs running while watching the System Usage window. I can repeat going from 100% CPU spikes with audio drop-outs to 0% CPU usage just by changing processor affinity under controlled circumstances.

Could you share where the settings are and what changes you made to achieve the results?

[59.9hz]

Thanks for the reminder about Hyper-threading... I've experimented with that as well, and came up with inconclusive results. What I'm on to now is a bit more specifically related to dynamically altering the resources that are available to Pro Tools, and this must be done while its running.

As for sharing the details, I'd be very happy to. But it might get a bit lengthy. I'm digging under the hood to play with some fundamental aspects of multi-threaded application tuning, which is a bit of a science mixed with chance.

There are a lot of factors in play, such as enabled plug-ins (running or not), disk and memory resources for Pro Tools, as well as consideration for all of the services in play such as the new collaboration features (and networking in general) the Application Manager and the communication between all of these. And don't forget Windows itself, and all of the process running that it needs. (wants)

What I'm doing is a bit like firing a bullet into a stuffed teddy bear and trying to figure out what is in the inside by examining the fluff that comes out the backside. In fact, I'm not too sure that I'm not affecting the stability of Pro Tools by doing this. Although I have observed very positive results so far (no crashes, errors, or drop-outs) I haven't tested this with lots of tracks and various plugins loaded.

I'm testing by using a facility in windows to set process priority and processor affinity. This can be done from Task Manager (Details tab, right-click on ProTools.exe) However, I've written a program so I can alter the values from within a batch file. Now I can use my work-around just by clicking a different Icon to start Pro Tools.

One of the things I've observed is that Pro Tools explicitly takes control of the priority and affinity when it starts. This means you can not alter these things upon launching Pro Tools. You can use the START shell command to start a process and specify these settings, but as soon as Pro Tools starts, it takes over and resets them to whatever it wants (all processor cores, normal priority.)

The other things I've observed Pro Tools takes control over is MMCSS (Multimedia Class Scheduler) and other miscellaneous things like the Visual Effects settings in Windows. I'm sure much of this tuning is due to the need to provide a consistent running environment to help minimize performance issues.

Why have I gone through this much trouble? Because I've been building computers and running different DAW software for about 15 years, and never have experienced the kind of CPU spikes and drop-outs/errors that I'm getting with Pro Tools. Even the older versions of Pro Tools weren't giving me this much trouble. With 12.5, I started to get CPU errors that stopped my tracking. When you are tracking live (and playing an instrument) and turn around after the take and realize that your DAW stopped recording halfway through, you get motivated to fix it.

As a software developer, I've created many multi-threaded programs, and it can really get complex... especially if you allow third-parties to build plug-ins that your software must yield to. But what I'm observing with Pro Tools is similar to threading race conditions I've seen with my own work. So, I have a bit of patience. However, with what I've observed, this seems like a problem that has been around for awhile, and gets worse/better with different revisions. Different order of operations or timing differences in the code could explain this.

I have a lot more details on exactly how I'm testing, but nothing I can do would compare to someone who could test with a proper profiler.

To test, I needed some way to monitor process times like interrupt requests and deferred procedure call (DPC) latencies. Also, I needed something that I could use to make the problem get worse on demand. I found the solution to both in one piece of software: LatencyMon. (Search Google) The newer version is even compatible with Windows 10. The standard test (just press go) will hook into the kernel and report some pretty precise timing of the afore mentioned items, plus tell you a bit about hard page faults and other performance aspects of your system.

I thought I was just going to observe what goes on when the CPU spikes show up, but when I started LatencyMon it made Pro Tools go bezerk. Nothing that I could find (process timing wise) correlated to this. Just the System Usage window showed the Total CPU off the scale red in consistent pulses, every second or two.

Then I experimented by going to Task Manager and setting the ProTools.exe process affinity to not use the last two processor cores. (I'm running a hexacore i7 with 12 virtual cores.) This left Pro Tools with only 10 cores to play with 2 cores free to do other Windows tasks. Immediately, the CPU went to zero, and Pro Tools was happy.

I left LatencyMon running in the background and did a couple hours worth of playing around with Pro Tools, recording, playing back, adding heavy-duty convolution plug-ins and VI's and didn't have a single problem.

Now, I leave LatencyMon out of the equation (I don't use it), and just start Pro Tools, then knock the process affinity back to 10 virtual cores, and I haven't seen a spike since. But if I repeat the initial test, I can cause spikes on demand.

I need to point out that most of the initial testing I did was when NO SESSION was even open. I would simply start Pro Tools and open the System Usage window and watch the chaos. When I repeated the test in a basic session (1 track) I could cause audio dropouts.

For now, I think I have found my work-around. I just hope it lasts through the next version release.
Yes, I'm a bit un-hopeful that a fix is in the works, because threading issues are tough to solve, and the fact that these CPU spikes have been around for awhile seems to suggest its something fundamental.

There's a whole bunch more testing I could do but I really would like to see if other Windows users have similar results just by using the Task Manager to mess with processor affinity. No need to download any tools or software. If you are having CPU spikes or CPU errors, you'll know it.

[TAMBOR]

I saw the same periodic CPU spike behavior when using LatencyMon, spikes reported by Protools system Usage but not by Windows Task Manager, if I recall correctly.

Have you tried setting Protools affinity to not use the first core(s) instead of the last? Or does it matter as long as it can't use all cores?

Do you think Protools is (mistakenly) trying to use all cores and conflicting with other system processes? I have read Windows like to pile system processes on the first core. There were some gamer forums where this was discussed, and there are guys there using utilities to control affinity for stuff like flight simulators.

Also with regard to hyperthreading, I read that on my CPU, the 5960X, there is still a lot of shared cache between the physical core and the virtual core, and so I question if there is much of a performance boost because of cache swapping. Do you have an opinion on that?

[jjnssn]

A couple weeks ago I used msconfig and dropped the # of hyper cores from 12 to 9 on my system. 9 seemed to be a sweet spot and got similar results.

I was able to run a 88.1 session at 128 buffers with no noticeable CPU peaking in PTs usage window with my HDX system.

I believe the issue is something avid is aware of given their recommendation on HDX systems to avoid the lowest buffer settings by sample rate.

So it is clear I am not running this way at the moment . What I did makes 3 cores not available. Windows monitor will also show only 9 cores.

I would be curious to understand what this does for others if they try.

http://www.sevenforums.com/tutorials...ndows-7-a.html




Sent from my iPhone using Tapatalk

[59.9hz]

Some terminology might be needed...

Socket = what you plug your chip into on your motherboard (Usually 1 of these, but some workstations have 2, servers can have more.)
In my case, I have 1. Most people have 1, so this is not usually a discussion topic, however at some point I may test with more than one on a workstation just to see what happens.

Core = One physical execution unit. In the old days, that's what was called a CPU, because nobody had dual/quad, etc. The i7 I'm testing with has 6 of these.

Thread = A series of work units that each can not be divided into smaller parts. The smallest unit of work possible. It is implied that the work unit can not be split/shared because it is indivisible. Let's say you wanted to count to 10. This would be done on one thread because you couldn't have another thread help you out, or you would lose track of the count. (That's sort of oversimplified, but you get the point..)

Process = A grouping of threads belonging to the same owner. For example ProTools.exe is a process. You can get a listing from Task Manager's details tab. (It shows the visible running processes only - yes there are others that aren't displayed.)

Now Hyper-threading... This is Intel' implementation to make a single core appear as two, implying that a single core can execute two threads, where only one would normally be possible. On the i7 I'm testing with, it shows up in Windows as 12 Logical Processors. (6 physical * 2 because of Hyper-threading = 12 [hyper]threads.) This is done primarily by doubling up on some structures in the processor core so it can grind away at two threads at once...but only superficially. The cache and other structures can be shared, and only certain operations can be performed in parallel. Intel pulled hyper-threading for some of its processors, and then re-introduced it when they revamped how it works. The i7 I'm using has the more advanced flavor, and performs better than the earlier versions. Your Mileage May Vary.

Now you start to see the problem. How does a program split its tasks up into 12 pieces that have to do things like share hardware (i/o device) and memory (waveforms) and still keep track of each other? Add to the complexity that all the plugins need to do their computations as well. This all needs to be coordinated. The term for this is Multi-Threading, i.e. 12 threads trying to execute and coordinate at the same time.

Coordination is the tricky part. Usually one thing finishes before something else, and that puts one thread in a Wait State because it can't do anything else because its waiting for something else to complete. For example, you might need to read a waveform from disk before you output it to the i/o. If these are split into two threads, you can see how they need to be coordinated.

What if you had thread 1 waiting on thread 2, and thread 2 waiting on thread 1? That sounds impossible. But stuff like this happens and must be dealt with. It all gets pretty complex and there are rules which must be followed to keep everyone playing together nicely. And that's only talking about one Process!

What about the other Processes? Lets not forget Windows itself. It needs to paint the graphics, move the mouse pointer, stream Pensado on YouTube, etc. ...funny, Pro Tools doesn't seem to crash when I do that :)

The term for sharing processor resources across multiple Processes/Threads is called Scheduling. This is performed largely by the Windows kernel, and it attempts to give everyone enough time to do their jobs. I.e. it switches the processing power between ProTools, Windows, other programs and background services like the AvidProjectSyncService, etc. B.T.W. Microsoft continually messes with how Windows does this. Different versions of Windows behave differently in how they choose to schedule threads. Windows 7,8,8.1,10 all exhibit slightly different behavior as Microsoft continues to tune to match more modern hardware as it evolves in the marketplace.

If any one of the above gets greedy and hogs too much time, or gets lost and has to move to the back of the line (re-scheduled) then someone is going to have to wait. If that someone is waiting to output to your i/o long enough to deplete your output buffer... well then you'll have a drop-out and Pro Tools might stop with a CPU error. This can (and often does) happen when Windows reports that your CPU's are at idle, or not working very hard. This is counter-intuitive and confusing to the user, who can clearly see there is plenty of unused CPU power! Theoretically, this could happen in less than .0007 seconds for one channel if that channel was using a 96k @ 64 buffer.

This is exactly the kind of thing that happens at a very low-level when you run a program like LatencyMon. Being a very low-level program itself, it has no qualms about demanding critical time from the CPU. At that low level, it competes directly with Pro Tools causing it to freak out. And yet Windows will happily report that the CPU's are not working very hard.

But this only seems to happen when Pro Tools is executing across all available virtual processors. (In my case all 12.) If we tell Windows to prevent this by setting processor affinity, Pro Tools instantly calms down, and the System Usage window shows this. (And audio no longer drops out.)

The magic number for me seems to be 2. I need to disable 2 virtual processors (just for the ProTools.exe process) to have a consistent stable happy Pro Tools. It can be processor 0 and 1, or 10 and 11, or some other combination, it doesn't matter.

A word about msconfig. This tool allows you to instruct Windows to only use a certain amount of logical processors after the next reboot. This feature is mainly used in debugging software, and simply instructs Windows to ignore some of the processor cores/virtual cores. This does not disable Hyper-threading. Hyper-threading is enabled/disabled at the motherboard firmware level, i.e. from BIOS.

As for HDX and other external DSP solutions, I would think that the above would apply as well. The role of the CPU in the computer is significantly diminished because it doesn't have to crunch the DSP algorithms. But there is still a great deal of i/o taking place. Also, it wouldn't surprise me if Avid took a different approach to buffering with HDX (differences in the audio engine between HD and Vanilla?) I don't know, I don't have HDX. I have used an RME RayDat to track 32 channels at 48K/32sample buffer though, without problems. That was on Pro Tools 11 something though, and this research is with version 12.5. Currently I use DigiGrid for all my tracking and overdubs.

So, all of this is pretty interesting, but very wordy to explain to this level of detail. I doubt many people made it far enough to read this sentence. Still, I hope this inspires people with CPU errors to give affinity a try. Don't use this on a production system until you have tested it to your level of comfort. I'd hate you to go through the same embarrassment I've gone through with people upset at having to redo takes.... But that was before I started down this testing path in the first place.

As for Avid, perhaps someone might read this and consider bringing back some form of CPU control to the Settings. Or just fix the spikes.

[TAMBOR]

Originally Posted by 59.9hz

But this only seems to happen when Pro Tools is executing across all available virtual processors. (In my case all 12.) If we tell Windows to prevent this by setting processor affinity, Pro Tools instantly calms down, and the System Usage window shows this. (And audio no longer drops out.)

The magic number for me seems to be 2. I need to disable 2 virtual processors (just for the ProTools.exe process) to have a consistent stable happy Pro Tools. It can be processor 0 and 1, or 10 and 11, or some other combination, it doesn't matter.


As for Avid, perhaps someone might read this and consider bringing back some form of CPU control to the Settings. Or just fix the spikes.

So do you have an educated guess as to why removing affinity for just a coupe of virtual processors causes PT to stop getting into the spike mode?

[59.9hz]

In technical terms, my guess would be a dependency chain causing thread contention.

But with what? That is the subject of additional testing. I've exhausted many of the easy tests. Most likely it has something to do with the drivers and other software I have installed and/or 3rd party code. (plug-ins) The fact that others are experiencing some form of this as well (expressed in various terms across many other posts here and elsewhere) seems to indicate I'm not alone.

This could even be a result of over-tuning. If you design a program that fails to yield correctly, your program might suffer consequences as a result. I'm not suggesting that's what's going on, but something is different about Pro Tools. Dare I say I use S1 without issues on the same hardware?

I'm tempted to strap on a debugger and trace the execution. Without any symbols or other documentation, it would be totally experimental. I just don't want to install something that obtrusive on my system. Yes I could take an image backup first, but I just don't have that much spare time. (Considering I can now just 'flip a switch' so to speak, and eliminate the problem.) I'd rather be playin; know what I'm sayin?

This issue has been around for awhile to varying degrees. When 12.5 was released, it came with several other elements to manage collaboration. These services run in the background, and for the first time, Avid have put forth the expectation that people have their DAW connected to the internet. This implies network traffic. I don't have issues with networking because I can saturate the network with simulated traffic, and Pro Tools does not suffer. I'm not sure what else those services are doing though. (At least not in detail.)

Before I did any testing, I made darn sure that ALL of my drivers/plug-ins/etc. were up to date. I suppose I could try more tests, like change the Engine to use a different interface, move plug-ins to disabled folder, etc.

IdeaScale? I wouldn't know where to start.

Eh.. I've typed too much already and my head hurts.

[Rock'N'Peanut]

@ 59.9hz
I was able to reproduce the same results you had by fallowing your instructions. I'm still getting some slight spikes but no where near the ones I was getting. The spikes are around 50-60 CPU usage where as before they were at 100 CPU usage. I was playing around with CPU Affinity a little and it seemed that the more cores I disabled the better Pro tools worked. It's strange I have never had this problem with any other versions of PT until, PT 12.4 and I have been using PT from V7 till now.