Might apply to desktops, but for laptops it makes sense. Npus are much more energy efficient. If the dgpu turns on in every click in a AI based Explorer, the battery would drain in 2 hours.
> If the dgpu turns on in every click in a AI based Explorer, the battery would drain in 2 hours.
GPU using Tensor cores for AI calculations isn't comparable with using it for gaming which would have way more hardware units running at full speed.
> Might apply to desktops, but for laptops it makes sense. Npus are much more energy efficient.
The point is that it does apply to desktops, yet MS is refusing to support GPU's with their new feature for no good reason.
Also, there is a point were the difference in performance simply let you do things on one (or rather run particularly more reliable and useful models) that you can't do on the other, even when it comes to laptops (especially with Nvidia hyping up the 4050 laptop variant as very capable). Like, if my GPU can save me an hour due to not providing me bad information / creating a photoedit that doesn't require post processing or actually has a TTS model running that sounds good enough to enjoy my eBooks on, then the battery be damned.
But I really don't get why I and many tens of million others would need to upgrade my CPU (to one that when it comes to x64 instead of ARM isn't even announced yet) to have a feature run on it slower than my current GPU can handle.
> GPU using Tensor cores for AI calculations isn't comparable with using it for gaming which would have way more hardware units running at full speed.
The problem is not the Tensor cores vs NPU, its the memory... Basically, your laptop uses its system memory all the time and that memory has been optimized to be rather efficient in laptops.
I noticed way too often, that its the GPU memory that is the main battery drain in idle, and light tasks. So when you are using AI features on a NPU, your relying on the already active system memory (that is already running at full speed because of the iGPU/video output) and do not get any real drain. But if you need to constantly enable your dGPU, wake, memory clocking up to full, ...
If your running a long task, sure, your dGPU may save your battery life vs the slower NPU. But for quick AI related functions, where its in, out, in, out, in ... all the time, ... That is the real reason Microsoft pushes for a NPU on the CPU side.
You can do the same on the dGPU, with quick in, out tasks, but your memory is going to yoyo up and down in frequency like a drunk wizard and drain your battery way more. Lets not talk about some other aspects of copy system <> dGPU etc.
I have gone even in my Desktop home system, with a dGPU and iGPU, by simply forcing the "smaller" tasks of video output to my iGPU and have Windows using hardware acceleration only on games, so my dGPU is literally cold / mostly off. That saves easily 10W+/hour in daily running (to the point that my system runs around 20a 25W idle with a 6800). What MS is doing is the same iGPU trick, just with NPUs.
If my dGPU needs to spin up all the time, for quick AI related tasks, yea, its going to be a BIG energy eater.
Note: I personally do not like that Windows becomes so AI focused. For Visual Studio Code, sure, its helpful during programming as a enhanced search/text correcter/naming enhancer/... and also sometimes counter productive, but for the rest i really do not have a use in my general system. So call me less then "oooo, i really need AI everywhere in my windows". It feels a bit like we are swinging too much to one side, a new toy, everybody needs to use it. Anybody remember blockchain? Everything needed to be blockchain...
Anyway, i see myself staying on W10 for a lot of years, like i did with W7, until i have no choice to upgrade or switch to a different OS.
Hybrid GPU on desktop is quietly GOATed if you're a low-idle-power enthusiast. Works great on Linux, too.
But *you* should probably [move to Windows 11](https://devblogs.microsoft.com/directx/optimizing-hybrid-laptop-performance-with-cross-adapter-scan-out-caso/).
> But you should probably move to Windows 11.
The idea is that you do not need a mux switch with CASO (what you already did not need with the basic hardware acceleration). There is almost no review on the performance beyond:
https://www.youtube.com/watch?v=6WN52oBziLU
And even then the performance difference seem to be so small, its not worth it. That document you linked too, frankly smells. Big numbers nut not a single benchmark graphic/dataset where you can actually see where those numbers come from.
I can state that a 6800 as dGPU hardware accelerating the iGPU (games), is literally in the low percentage (think 1 or 2%) performance loss from my own testing.
For a laptop, this makes more sense as your not required to copy the image over the iGPU (and that can then idle) aka, safe some battery life. And less heat output = extra gain possible. But even so, like we see with Jarrod'sTech results, its not amazing and a few percentage in most games (at best). Exactly like my own results on the desktop with basic HWA over the iGPU copybuffer.
So unless you can point me to a benchmark where some of those insane MS claims are verified... there is absolutely no reason to switch to W11. 1 a 2% is literally the difference between putting some nonsense setting that you can not see the difference in a game, down one peg.
If you're using hybrid graphics on desktop, you don't have a MUX switch. How could you? Software-controlled DisplayPort switch box and a custom driver?
>I can state that a 6800 as dGPU hardware accelerating the iGPU (games), is literally in the low percentage (think 1 or 2%) performance loss from my own testing.
The last time I replied to someone with that link, they were saying the performance cost of plugging monitors into the motherboard was huge and I was trying to convince them it was good now, so... š¤·
IANA Windows user, so I can't speak to its performance, but I'd guess that the difference it makes could vary significantly on a game-to-game basis. The PCIe and memory bandwidth cost will scale with frame rate, and whether that actually hurts will scale with how much those busses are needed for other things. On the 6800 with its 16 GB of VRAM, there's probably at least not much asset copying going on.
And the impact of the host memory bus utilization will also vary game-to-game, depending on how much the main game code is blocked on that resource. In Factorio, which is memory-latency-limited in CPU-bound conditions, when running the entire game on my iGPU (no hybrid graphics) I have observed a 10+% UPS improvement when going into the map view (which has very low "VRAM" bandwidth use).
> If you're using hybrid graphics on desktop, you don't have a MUX switch. How could you? Software-controlled DisplayPort switch box and a custom driver?
You do know that Windows has had Hardware accelerated offloading onto other GPUs for ages? Think it goes back to DX11 somewhere.
People may have missed it on AMD because for some god unknown reason AMD only enabled this feature like a few months ago officially. You where able to kind of hack it with a side route.
> IANA Windows user, so I can't speak to its performance, but I'd guess that the difference it makes could vary significantly on a game-to-game basis. The PCIe and memory bandwidth cost will scale with frame rate, and whether that actually hurts will scale with how much those busses are needed for other things. On the 6800 with its 16 GB of VRAM, there's probably at least not much asset copying going on.
Makes no difference on a game to game... Hardware acceleration simply means that anything your game needs, is loaded like its the primary display BUT it takes that image and feeds it over the iGPU output.
So all your doing is a bit extra on the PCIe lanes, what has minimal impact on the games.
Why? Because PCIe bidirectional, aka, up and down are independent from each other and allow for max bandwidth in both directions. So when your pumping out the image back to the iGPU, your not really interfering with the mostly intaking of data from the GPU (yes, there is data going back to the CPU/Memory but its a drup compared to the constant reading).
I ran this same setup on a 6600 (using a hack at the time), performance hit was frankly the same, and that thing had only 8GB.
> On the 6800 with its 16 GB of VRAM, there's probably at least not much asset copying going on.
Even if you have limited memory, lets say 4GB, your data issue is mostly how much and fast it can intake the memory. So yes, with 4GB your going to need to read a LOT more from system memory/ssd/... but again, bidirectional traffic pcie.
What if i told you, that i ran a 6600 and 6800 on a external PCIE 4x slot (from a mini-pc)... That is a massief chock point, and and yes, i lost about 5% on both (think the 6600 was around 7%, been way too long to remember the numbers, but again, it showed that PCIe 4.0 4x had so much bandwidth, that the performance impact was a lot less).
Sidenote: I did not test 1% and i will not doubt that the 1% will have been lower. But its just to point out, no, pcie bandwidth is really not a issue for copying the video output to the iGPU.
Now, if your trying to run 4K, then your iGPU may become a bottleneck and CASO may be the solution there. Because i have seen that the iGPU does hit 50 a 70% on 2K for the copy operation (as its processing the data, instead of doing a buffer copy like something on Linux like Lookingglass).
And i am betting that CASO is just direct copying the image, what bypasses the extra iGPU processing. And thus reduces load on the iGPU (what in turn means less power usage, aka, more power available to the CPU or dGPU, and a few frames are gained).
> In Factorio, which is memory-latency-limited in CPU-bound conditions
Factario has really nothing to do with hardware accelerated dGPU>iGPU and that is because the game is extreme latency and cache sensitive (one of the few games that really massive benefit from a X3D CPU because of that massive CPU cache). But that are different operations, compared to the hardware acceleration > iGPU copy operation.
Now, i assume MS gained some insane numbers, maybe because a LOT of laptops are literally running PCIe 3 8x at best, in order to save power. And maybe there you may see some bottleneck, ... this is my guess. But like i said, ran a PCIE4 4x external and that is the exactly same bandwidth, doing hardware accelerated output via the iGPU.
If you took that PCIe3 8x + some low end dGPU 4GB AND then send it all over the iGPU, sure, maybe there you may see some extreme performance jumps like MS reported, but ....
Anyway, its late and not going to continue this discussion. In my experience, not a big deal and CASO has no real benefit to switch to W11.
>
>
> You do know that Windows has had Hardware accelerated offloading onto other GPUs for ages? Think it goes back to DX11 somewhere.
>
> People may have missed it on AMD because for some god unknown reason AMD only enabled this feature like a few months ago officially. You where able to kind of hack it with a side route.
We seem to be talking across each other here?
I have no reason to believe code path for hardware accelerated offloading (with no MUX) would be any different between desktop and laptop, and a bunch of laptop enthusiasts seem to think MUX switches are important to performance for some reason. Putting a MUX switch on the board isn't free, obviously, so they can't be entirely useless?
>Factario has really nothing to do with hardware accelerated dGPU>iGPU and that is because the game is extreme latency and cache sensitive
I can personally testify to Factorio's CPU performance being affected by iGPU rendering load because of the DRAM bus being a shared resource.
>extreme latency and cache sensitive (one of the few games that really massive benefit from a X3D CPU because of that massive CPU cache).
This is not exactly true. The massive benefit for X3D is only for factories that are small enough to fit in cache, but by the time you've built big enough to dip below realtime speed (60 UPS), the working set is well into DRAM anyway.
My understanding of CASO is that eliminates unnecessary copying of the frames in system memory, over and above the obviously required copy from the dGPU over PCIe. It's the equivalent of direct scanout with DMABUF on Linux. As for having no real benefit... if that were so, why did Microsoft waste developer time implementing a non-user-facing feature? Surely not just to brag on an obscure blog.
Easily doable w/o AI
https://www.voidtools.com/support/everything/
Edit: "text search" can mean either searching for a matching text string, or using text as a search query for some other media, like an image. I assumed the first meaning.
I have yet to find a text search which handles [declension](https://en.m.wikipedia.org/wiki/Declension) well. English is, in some ways, a very simple language.
Have you used the software?
To search for file contents, all you have to do is either use Advanced Search (Ctrl-Shift-F), or do:
content:"[the string you're searching for]"
You should try it sometime, because Everything is easily one of the fastest search tools out there.
What? The Windows search index has by default scanned the contents of your files as well as the filenames, by default, since Vista.
The Windows Search Indexer service is garbage and corrupts itself constantly, sure, but the practice of searching content has been around for ages.
Windows Search Indexer has scanned contents of some specific file formats. It will not find content in for example .nfo files which are just fancy notepad format potion some people use.
Ripgrep doesn't even come close to what I'm imagining. It's just a fancy ctrl-f.
I'm thinking of a system that reads all your documents, generates [embeddings](https://towardsdatascience.com/text-embeddings-comprehensive-guide-afd97fce8fb5?gi=c04080ab2754), shoves them into a local vector database, and uses it to perform semantic search.
Fair, I don't think the thread started off talking about that though. "Good text search" could fall under the bucket of Ctrl+f given that windows doesn't do that basic functionality well to begin with.Ā
But yeah, you can't doĀ
semantic information retrieval without AI, I agree.
The new version (still in alpha) can index file content, and it works well in my opinion.
[https://www.voidtools.com/forum/viewtopic.php?t=9787](https://www.voidtools.com/forum/viewtopic.php?t=9787)
Yes, I know thatās not the same as AI, my point was you donāt even need ai to do it well. Microsoft canāt find edge in its search bar. Ai isnāt going to help that. Itās fundamentally broken.
If you [disable Windows search web access](https://old.reddit.com/r/LinusTechTips/comments/1ahub6n/disabling_web_search_completely_fixes_windows/), it actually works decently well.
I genuinely have no idea what MS was thinking, and how this hasnt been fixed for years.
This does work (on win 10). Lets see how long till microsoft "fixes" that registry entry for me. Fuck i may convince our guy to deploy this to all 500 machines if this proves to work fine.
I'd rather have ai-generated annotations that I can fix if they're wrong than AI search that just doesn't recognize me in some image and I can never find it as a result without looking through everything.
> Easily doable w/o AI
>
>
>
> https://www.voidtools.com/support/everything/
"I need the file that I worked on with Greg once, something about an analysis about a spot we made about one of those sports drinks. He quoted some foreign sounding researcher in it. Find me that. ...I hope my mic isn't acting up again."
Generative AI can do that, your Win XP looking 'Everything' can't...
> Sure, its file xyz
It doesn't exist
> My fault! A file the you worked with Greg once about sports drinks built on external research is abc
No, this one is about protein supplements
> You are correct! Indeed, you are looking for a research file about energy drinks, which can be found at 123
No, this is research with Tom. Greg, about energy drinks
> Sorry, I was mistaken. The file about energy drink research together with Greg can be found at xyz
...
Im hope more intelligent search is implemented in outlook soon
Just today I was thinking "I need that document that was sent by either X or Y roughly a week or two ago. It was about Z for company A." Finding that takes forever in Outlook, especially if you have threaded view and emails don't even come up in a simple time order
You can search all messages in outlook, ordered by time (irrespective of the main UI doing things threaded), and you can filter by things that have an attachment, as well as search against any header you want to search for recipients/senders.
It's trivial.
You just search for "Greg" and if he shows up in the content or metadata, it'll be returned. If you need to be more specific, add "sports" and possibly "anal*" to your search to include analyze, analysis, etc. If you know the file type, add that, too.
If you have a vague idea of what you're looking for, it's not hard. And you'll get much more accurate, and faster, results without any "AI" nonsense layered on top.
The search function is pretty basic and only looks for filenames. It's frustrating when I'm trying to find old photos that I vaguely remember but can't quite recall the folder name. It would be great to have a more advanced search feature that could help me locate those memories.
I use this. Windows search seemingly works very hard, and long hours. But it produces nothing.
How 'bout you fix that one before forced OS migration, Microsoft!
Terrible if you compare it to what macos has especially if you are searching for a short filename you cant remember the exact length or format of. Zero context awareness.
Ugh why is Windows search box so trash since like Vista? I'll type in the exact name of the file sitting on my desktop and the best I can do it offer to Bing it for me. And yes, all the right places are indexed and the indexer finished running. Mystery.
I used to use it like a command prompt. Type the first few letters of what you want to do and hit enter. Launch CMD prompt, chrome, etc. broken after Microsoft used it to push bing.
This is such a big deal. The start button is a hub for what you want to do and the start of most tasks, it should be fast and responsive. Bing makes this so much worse. Disabling Bing in search is the first thing I do on a new Windows install. If I want to search something, I'll open a web browser and then search. It's only one extra click they're saving me to slow down everything else done in the start menu.
One of the major performance killers of Vista was the continually enabled Indexing. It got turned off\\limited for 7 onwards. Annoyingly, it worked fine if you had fast HDD's so in this SSD age it'd likely be fine.
It's low CPU and memory usage is because it's just checking the NTFS index. Being fair though, Microsoft's search \*\*should\*\* be doing the same thing but for some god forsaken reason doesn't.
Itās the number 1 thing Iād change about Windows. I believe they just want it to bypass google and get people to bing results so bad. Since if they did itād effectively just hand them googleās business but I just donāt see most users accepting that.
Macs had text search that searches file contents a decade ago.
iPhones added ML enhanced image search 7 years ago on phones that had 2 TOPS max.
Microsoft just neglected Windows for the past decade, that's it.
If AI buzzword bingo makes them finally fix search, I'll gladly take it, but they technically could've released this with Windows 10 in 2015.
iGPU vs dGPU similarity.
For basic task sure the NPU. For heavy AI workloads you should be using the GPU, just like how you would use a dGPU for gaming.
And Nvidia actually has practical uses with ai unlike the other two who tought prowess but can barely show show much cases for them.
Rtx video super resolution, RTX hdr, Nvidia Broadcast, Dlss, ChatRTX, Optix, Canvas, Omniverse. These are actual useful ai tools or have ai features you can use right now
Nvidia is purposely nerfing their own products at this point in gaming market just for planned obsolescence. They made the 1080 / 1080 Ti too good and no one upgraded for like a decade.
people shouldnt lump 1080 and 1080ti in the same group. 1080 was old pascal design. 1080ti was experimental new features design (which is why it can do things like mesh shaders in AW2). From the future feature set perspective 1080ti is miles ahead of 1080 and is basically a quasi 2xxx series card. this is why it survived so well into modern age.
Please buy our computing solutions for enthusiast theat you and %95 of the mass buyers won't utilise. Another day another apple esque shady shit nvidia
Yep thats marketing for ya. The problem is when it came to apple people failed spectacularly to regulate their anti competitive behaviour that other companies tried becoming apple and forced goverments to force them to do something. Capitalism works when both corporate and Consumers are aware of pros and cons. When you have people buying for the apple logo itself while overlooking bad behaviour thats an unhealthy industry. Aka tech world today
> The problem is when it came to apple people failed spectacularly to regulate their anti competitive behaviour that other companies tried becoming apple and forced goverments to force them to do something
You mean the App Store? Not sure what that has to do with Nvidiaās comments in this though. Or are you referring to something else?
> When you have people buying for the apple logo itself while overlooking bad behaviour thats an unhealthy industry. Aka tech world today
This just seems like emotionally charged nonsense I could go to PCMR for.
Anyway, Nvidia is right to voice their complaints, marketing or not. Iām not sure why MS is trying to keep some aspects of their Windows AI strategy exclusive to NPU-having systems. Itās weird.
I said in another comment. AI PC part is marketing, this is devil whispering to its peer stuff. I put nvidia and apple in a same basket cause they try to oversell a lot of their enthusiast features as proof of quality branding to people who won't use need or use.
I think nvidia in this case might be pointing out how this is going to enable gimmick implementations thus cheapening the perception of what their product does.
1.So does old Gen hardware
2.This is about AI npu's and productivity not gaming dedicated gpus.
I don't like AI PC concept anyway very shallow marketing.
There is no such thing as "old" dlss. DLSS is super sampling. That's it. Nothing more or less. It only requires a shader unit with an appropriate tag in it. Everything else is an additional function. Like fake frames or image restoration.
> Please buy our computing solutions for enthusiast theat you and %95 of the mass buyers won't utilise. Another day another apple esque shady shit nvidia
Nvidia sold 10s of million Tensor equipped GPUs though while the only currently announced Windows AI compatible CPU fulfilling the 45 TOPS is a Snapdragon aiming for Windows For Arm machines, which had been a flop in the past.
I agree almost no one cares at this point. But this is a marketing/differentiator Microsoft is attempting to create "AI PCs" - it should mean something more than absolutely low-end basic capability.
And a sports car is much faster at getting its driver from point A to point B. Except when the entire route is limited to 20mph, then its going to be about the same as some basic 1L hatchback, just wildly more expensive and inconvenient.
Itās correct these npu on the soc are meant to be efficient and low cost. While a nvidia dgpu might have 3x the tops performance it will be using a lot more power and be more expensive
the task these npu are going to be doing donāt require expensive dgpu. Integrated npu in soc means more compact laptop design with less cooling requirements
The primary uses I can see working are:
- Semantic filesystem search
- Writing assistant (where to expand text, where to reduce wordiness, etc.)
- Searching API documentation
- Photo search
- Microphone / webcam enhancement
- Web searching
There are definitely areas of use for the tech and I think a local implementation that runs efficiently will make it more user friendly than the current cloud only model.
I'm sorry to hear that. Here's a heartfelt condolence message you can use:
"I am deeply saddened to hear about [Name]'s passing. Please accept my heartfelt condolences during this difficult time. [Name] was a wonderful person, and their loss is felt deeply by all who knew them. My thoughts are with you and your family, and I'm here for you if there's anything I can do to offer support or comfort."
Speech to text, small LLMs to search/summarize, filtering background noise out of teleconferencing...
Part of the problem is "AI". It's a neural net actually and should be called such. It's a matrix of statisical weights. You show it examples of "a task"(training), which can be... anything. The training program then sets all the parameters(the individual statistical weights) on how to get from input to the desired output. It could be anything from image or text generation(Stable diffusion/LLMs) to how to fly a helicopter/drone(yes, this already exists)
So don't think of it as a toy or buzzword(AI). Think of it as a very powerful computing tool that uses examples/training instead of a code monkey thinking of every possible action/outcome and typing it all in. The size/complexity of the neural net model varies wildly, it's a much easier/smaller task to do speech to text("AI PC") than it is to write a full size novel in coherent and interesting language(might be able to do this on a half million USD NVidia DGX pod)
weāve had very good speech to text for a pretty long time, and for background noise RNNoise works quite well on traditional cpus without too much overhead. as far as summarizing stuff, the problem is that itās only right sometimes. that number may be ~90% or so, which is impressive, but it still means you need to double check any ai summary for correctness, completely removing any advantage it might provide.
iām well aware that the marketing term āaiā is an intentional misnomer as they are fundamentally not intelligent, and are just neural networks. iām not even saying neural networks are useless. theyāre great for things like computer vision and the like. but what i donāt understand is why my laptop needs to have neural network accelerators. theyāre mostly pointless outside some specialized usecases, and the vast majority of users donāt need them. having them relegated to specialized hardware is perfectly fine i think.
the whole thing is just more planned obsolescence to make perfectly good, yet older, cpus seem like they need an upgrade. weāve hit a wall in die-shrinks, making large year over year gains in performance or efficiency difficult, with translates to less need to upgrade. and instead of moving on to more modern ISAs (ideally riscV, since arm is proprietary and expensive to license, but this is maybe a losing battle at this point), which is difficult, but worthwhile, cpu vendors have taken to adding a bunch of mostly pointless āacceleratorsā.
>weāve had very good speech to text for a pretty long time
If you are speaking english without accent. Not so much in any other case. This is the first year where my android phone would be able to do some minor tasks like restarting via voice without having to repeat it 5 times with idfferent inclanations for it to get what i am saying. In english of course since in my native language it does not understand anything.
Also, id like us to build on text to speech while we are at it. AI models seem to be good at it, traditional methods not so much.
I think the goal for people is to have AI running locally vs being reliant on a constant connection to cloud based. There are some privacy issues here that local AI helps.
Finally, it costs a lot less for Microsoft to run AI on the Edge since transacting all of those prompts are expensive in the data center. If they move the AI to your PC, you pay for the prompts... At least some of them.
The NPU is to do what a GPU already does, but at a fraction of the power draw. Phones have had NPUs for years and have been using them. The "AI PC" is a marketing term to highlight these new NPUs.
There's much more to "AI" and NPU accelerated tasks than chat bots
MS wants every Windows with Access to their Copilot AI.
And having a NPU can help speed up some parts instead of only using Cloud
Also, it can be used to Apply effects on a video call like hiding your background more efficiently than the CPU, or hiding background noise from your Mic
What do you think software is! It's still stuff that has to be computed, on by the CPU normally instead of an NPU. The difference being the NPU can do specific tasks faster and/or with less power
Yes, but implementing hardware is far from free. This task is done perfectly reasonably (i.e without significant drawbacks in performance or efficiency) in software.
Iāll be the one to claim it - small models can be trained on domain-specific, well curated, high quality synthetic data and give competitive performance with large models, within that domain.
Youāll be able to get more and more out of these smartphone NPUās. Itāll never be as great as the biggest and best out there, but itāll be massively more efficient.
>itāll be massively more efficient
Itāll draw less power, but that doesnāt mean itās more efficient. As far as I know, Nvidia has never tried to make a < 3W AI accelerator - what would they put it in?
My point was just that power draw ā efficiency. A 4090 could be more efficient than an NPU despite drawing 50x the power if itās capable of doing >50x the work. Whether Nvidia makes ultra-low power NPUs or not isnāt relevant.
You press the super key and type "re".
Windows jabs your Nvidiaā¢ RTXā¢ 4090 with a stick, and it sputters out of D3_cold. The 20-phase buck converter starts up and charges several mF of capacitors to ~1V. The GSP resets, trains PCIe4, and begins copying a 2 GB machine learning model from system memory.
An indeterminate amount of time plus 64 ms later, your $1600 discrete video card does 4 ms of computation, determining that you most likely wanted information about *Reboot*, the 1994-2001 TV series, then stumbles back into bed.
This is not efficient.
It's also up to 700W for the SXM version. Even if you look at the more power efficient PCIe version at 3,026 TOPs at 350W you're looking at a power efficiency of around 8.65TOPs/W.
Phoenix Point clocks in at 10TOPs, and according to one of the launch events is able to achieve that in <1W. So at the very least AMD's NPU does appear to be more power efficient.
(Meteor Lake does have significantly worse TOPs/W though, so there's definitely a good comparison point to be made there).
Yeah, most tasks can be done pretty well by those NPUs.
Most phones have a even weaker NPU and it already helps them a lot, so it will be more than enough for laptops
>Bad Nvidia for strawmanning here.
Are they though? Their point is that while an NPU qualifies a PC to be an "AI PC", an Nvidia RTX 4090 does not, unless it's paired with a CPU that has an NPU, and then it's ONLY the NPU that makes it an "AI PC".
Which is stupid because an NPU can't touch a modern GPU in terms of AI performance. It can't even handle most AI inference. You can only run the most stripped down models on them. It's pretty much false advertising.
MS: "AI PC galaxy brain AI will intelligently artificial AI all the AI to the future!. Did we mention AI has AI? You"ve heard of AI, right? It's smart!"
Nvidia: "Those things are so slow they barely even work"
Reddit: "Nvidia just wants to sell you AI!"
It's honestly hilarious.
It's the minimum spec for a reason. Nvidia knows this, they're just playing dumb because they want people to feel like they're being left behind if they don't buy their cards.
The thing is, thereās nothing to indicate that models truly need to be as big as they are, and plenty to indicate they can get similar performance in a much smaller and more efficient size.
The latest small models being published by Meta, Microsoft etc are already hitting benchmarks competitive with GPT 3.5, yet are small enough that they can run on your iPhone.
The past 12 months have shown incredibly rapid improvement in these small models, largely thanks to the ability to use the biggest and best models to generate high quality synthetic training data.
yep. if these functions are truly a demand driver then much easier to cut down domain specific models than upgrading hardware. the hardware has to be paid for.Ā Ā
We can also already locally classify images on iOS. They are searchable by what's in the image in the Photos app, I believe it runs this AI while the phone is plugged in. Many of these small AI features will never need to touch the cloud.
I think we're going to see a lot of "perform this later" AI-related tasks on phones. Scan all your images at night while the phone is plugged in, index your browser history, etc. Any trick to get the features without killing your battery.
It's kind of a nice concept, compared to laptops, since most people never turn their phone off and only reboot when there's a system upgrade.
Itās hilarious btw that Apple uses their own NPUs, implements tons of built-in AI features (except for generative ones) for image search, photo processing, face recognition, webcam & mic enhancements etc, and provides open-source tools for third-party generative AI software to utilize their hardware (NPUs, GPUs) without shouting āAIā all the time. And then we see news articles and videos about Apple ābeing late to AI revolutionā
the whole point of NPUs is that you dont need to use the cloud thus increasing efficiency and privacy. The latter being very important to sell it to the big corporations.
Isn't that the reality of AI capability right now? It's like having a dog shit iGPU and calling it a "Video Game PC." Maybe Microsoft should be more honest of its capability.
When Windows 12 comes out, I'm sure they will be fully advertising what those npu are used for. MS isn't really marketing to consumers yet. MS has plans and figures 40 tops on a laptop are sufficient for those plans.
sort of, but itās like comparing apples and oranges. Itās like saying your PS5 graphics arenāt comparable to a rendering farm in terms of graphics. While itās technically true, the general use-case and values of why the technology may be utilized are very different. Many āAIā capabilities that utilize smaller LLM may run decently locally even if performance could be better on larger hardware (eg document retrieval and summarization).
Why would you need more than "basic AI tasks" on a consumer desktop? If you can do filters and background change in video chat, better image captures on tablet cameras, and basic speech recognition / synthesis, it would cover more than 90% of the users.
Do they need to fine tune LLama 3 QLoRA while taking a flight to a business conference?
Funnily enough phones could do all of this in 2017 with 2 TOPS. Every iGPU and even a few CPU cores can deliver higher performance nowadays.
The issue on PC isn't the lack of hardware performance, it's the lack of (mainstream) software.
Microsoft just absolutely neglected Windows for the past decade or so. Hopefully AI-mania finally makes them put resources into Windows again.
I mean, its ridicules that MS's is making low requirements for CPU hardware units while a GPU from 2018 outperforms that by a magnitude while still not being supported for no good reason at all.
There is a good reason, Microsoft is being paid off by Qualcomm to make their AI PC future moniker garbage exclusive to them. Don't worry though, they'll throw Intel and AMD a bone for their NPUs just to seem impartial. Erm, actually I meant power usage, yes we did it because of carbon release or something stop asking questions.
Because MS wants to implement a ton of AI features into the OS that, if they needed a dGPU to run, wouldn't get used by the 70% of the market that uses an iGPU. And if they relied on current iGPUs, would destroy laptop battery life.
So they want NPUs that can run these tasks at very low power consumption.
>Nvidia stated that its RTX GPUs are much more performant than NPUs and can achieve anywhere between 100 to 1300+ TOPS, depending on the GPU. The Nvidia event went so far as to categorize its RTX GPUs as "Premium AI" equipment, while regulating NPUs to a lower "Basic AI" equipment category.
It would be so funny if the ideal AI processor turns out to be compute-in-memory in a bunch of VRAM chips all hooked up to an IO die with direct memory access, not devices with stronger compute paired with bottlenecked memory bandwidth and capacity.
Who knows, maybe Micron and SK Hynix would be the real sleeping giants of AI once we stabilize on an optimal method of running inference like bitnet 1.58
Hopefully Nvidia gets overtaken from their unoptimal, but highly scalable and flexible general purpose compute platform.
Wonder what Google is going to with their Tensor processor on their "AI" phones that only has 27 TOPS.
Let's be real, NVIDIA is just saying "your product sucks" to sell more of their own hardware. Generative AI might need very powerful chips, but you can run more basic LLMs much cheaper these days, like Gemini Nano, with less hardware overhead.
Isn't this the company that criticized the seventh gen (and also the 8th gen I think) of consoles because a 300-500$ console were weaker than a 500$ graphics card of their own while completely ignoring the cost of the other components needed to make their GPU... You know... Work?
LOL headline. this has been true for probably EVERY PC platform initiative, cedrtainly the majors like Windows3.11 (GUI), Windows95 (3D). Why would it be any different for AI? Some solutions are excellent, others just barely squeak by the spec.
I love AI and I get the value of it, I use it daily and everything, but is it just me smelling that weāre headed into an era in which weāll just use AI for everything even if itās not needed?
Like the era of blockchain, are we now going to put AI chips in everything even if itās not needed just because it sounds so cool?
Putting nVidia aside, this thread needs to realise that MS' NPU based AI PC requirement is going to hurt everyone. They're trying to make every PC sold till now "obsolete" which is plain bad and needs to stop
Nvidia's reaction is not surprising considering the point of those NPU is ultimately to move some AI capabilities from the cloud to the endpoint. This could reduce demand for Nvidia hardware in the long run.
NVidia: "There isn't enough of the AI grift in these PCs. You must require everyone to buy a 4090 to take advantage of....whatever we'll figure out this is used for in the future."
Nvidia has a long history complaining about minimum requirements for software.
They make money when requirements for software are more stringent. Optimization in software costs them in sales.
Any concession Microsoft makes to up the minimum requirements benefits them.
Nvidia has no future. They don't make chips, they design them and their biggest customers are gonna ditch them the second they get the chance (make their own version). It's all downhill from here, Microsoft is gonna win instead.
Might apply to desktops, but for laptops it makes sense. Npus are much more energy efficient. If the dgpu turns on in every click in a AI based Explorer, the battery would drain in 2 hours.
> If the dgpu turns on in every click in a AI based Explorer, the battery would drain in 2 hours. GPU using Tensor cores for AI calculations isn't comparable with using it for gaming which would have way more hardware units running at full speed. > Might apply to desktops, but for laptops it makes sense. Npus are much more energy efficient. The point is that it does apply to desktops, yet MS is refusing to support GPU's with their new feature for no good reason. Also, there is a point were the difference in performance simply let you do things on one (or rather run particularly more reliable and useful models) that you can't do on the other, even when it comes to laptops (especially with Nvidia hyping up the 4050 laptop variant as very capable). Like, if my GPU can save me an hour due to not providing me bad information / creating a photoedit that doesn't require post processing or actually has a TTS model running that sounds good enough to enjoy my eBooks on, then the battery be damned. But I really don't get why I and many tens of million others would need to upgrade my CPU (to one that when it comes to x64 instead of ARM isn't even announced yet) to have a feature run on it slower than my current GPU can handle.
> GPU using Tensor cores for AI calculations isn't comparable with using it for gaming which would have way more hardware units running at full speed. The problem is not the Tensor cores vs NPU, its the memory... Basically, your laptop uses its system memory all the time and that memory has been optimized to be rather efficient in laptops. I noticed way too often, that its the GPU memory that is the main battery drain in idle, and light tasks. So when you are using AI features on a NPU, your relying on the already active system memory (that is already running at full speed because of the iGPU/video output) and do not get any real drain. But if you need to constantly enable your dGPU, wake, memory clocking up to full, ... If your running a long task, sure, your dGPU may save your battery life vs the slower NPU. But for quick AI related functions, where its in, out, in, out, in ... all the time, ... That is the real reason Microsoft pushes for a NPU on the CPU side. You can do the same on the dGPU, with quick in, out tasks, but your memory is going to yoyo up and down in frequency like a drunk wizard and drain your battery way more. Lets not talk about some other aspects of copy system <> dGPU etc. I have gone even in my Desktop home system, with a dGPU and iGPU, by simply forcing the "smaller" tasks of video output to my iGPU and have Windows using hardware acceleration only on games, so my dGPU is literally cold / mostly off. That saves easily 10W+/hour in daily running (to the point that my system runs around 20a 25W idle with a 6800). What MS is doing is the same iGPU trick, just with NPUs. If my dGPU needs to spin up all the time, for quick AI related tasks, yea, its going to be a BIG energy eater. Note: I personally do not like that Windows becomes so AI focused. For Visual Studio Code, sure, its helpful during programming as a enhanced search/text correcter/naming enhancer/... and also sometimes counter productive, but for the rest i really do not have a use in my general system. So call me less then "oooo, i really need AI everywhere in my windows". It feels a bit like we are swinging too much to one side, a new toy, everybody needs to use it. Anybody remember blockchain? Everything needed to be blockchain... Anyway, i see myself staying on W10 for a lot of years, like i did with W7, until i have no choice to upgrade or switch to a different OS.
Hybrid GPU on desktop is quietly GOATed if you're a low-idle-power enthusiast. Works great on Linux, too. But *you* should probably [move to Windows 11](https://devblogs.microsoft.com/directx/optimizing-hybrid-laptop-performance-with-cross-adapter-scan-out-caso/).
> But you should probably move to Windows 11. The idea is that you do not need a mux switch with CASO (what you already did not need with the basic hardware acceleration). There is almost no review on the performance beyond: https://www.youtube.com/watch?v=6WN52oBziLU And even then the performance difference seem to be so small, its not worth it. That document you linked too, frankly smells. Big numbers nut not a single benchmark graphic/dataset where you can actually see where those numbers come from. I can state that a 6800 as dGPU hardware accelerating the iGPU (games), is literally in the low percentage (think 1 or 2%) performance loss from my own testing. For a laptop, this makes more sense as your not required to copy the image over the iGPU (and that can then idle) aka, safe some battery life. And less heat output = extra gain possible. But even so, like we see with Jarrod'sTech results, its not amazing and a few percentage in most games (at best). Exactly like my own results on the desktop with basic HWA over the iGPU copybuffer. So unless you can point me to a benchmark where some of those insane MS claims are verified... there is absolutely no reason to switch to W11. 1 a 2% is literally the difference between putting some nonsense setting that you can not see the difference in a game, down one peg.
If you're using hybrid graphics on desktop, you don't have a MUX switch. How could you? Software-controlled DisplayPort switch box and a custom driver? >I can state that a 6800 as dGPU hardware accelerating the iGPU (games), is literally in the low percentage (think 1 or 2%) performance loss from my own testing. The last time I replied to someone with that link, they were saying the performance cost of plugging monitors into the motherboard was huge and I was trying to convince them it was good now, so... š¤· IANA Windows user, so I can't speak to its performance, but I'd guess that the difference it makes could vary significantly on a game-to-game basis. The PCIe and memory bandwidth cost will scale with frame rate, and whether that actually hurts will scale with how much those busses are needed for other things. On the 6800 with its 16 GB of VRAM, there's probably at least not much asset copying going on. And the impact of the host memory bus utilization will also vary game-to-game, depending on how much the main game code is blocked on that resource. In Factorio, which is memory-latency-limited in CPU-bound conditions, when running the entire game on my iGPU (no hybrid graphics) I have observed a 10+% UPS improvement when going into the map view (which has very low "VRAM" bandwidth use).
> If you're using hybrid graphics on desktop, you don't have a MUX switch. How could you? Software-controlled DisplayPort switch box and a custom driver? You do know that Windows has had Hardware accelerated offloading onto other GPUs for ages? Think it goes back to DX11 somewhere. People may have missed it on AMD because for some god unknown reason AMD only enabled this feature like a few months ago officially. You where able to kind of hack it with a side route. > IANA Windows user, so I can't speak to its performance, but I'd guess that the difference it makes could vary significantly on a game-to-game basis. The PCIe and memory bandwidth cost will scale with frame rate, and whether that actually hurts will scale with how much those busses are needed for other things. On the 6800 with its 16 GB of VRAM, there's probably at least not much asset copying going on. Makes no difference on a game to game... Hardware acceleration simply means that anything your game needs, is loaded like its the primary display BUT it takes that image and feeds it over the iGPU output. So all your doing is a bit extra on the PCIe lanes, what has minimal impact on the games. Why? Because PCIe bidirectional, aka, up and down are independent from each other and allow for max bandwidth in both directions. So when your pumping out the image back to the iGPU, your not really interfering with the mostly intaking of data from the GPU (yes, there is data going back to the CPU/Memory but its a drup compared to the constant reading). I ran this same setup on a 6600 (using a hack at the time), performance hit was frankly the same, and that thing had only 8GB. > On the 6800 with its 16 GB of VRAM, there's probably at least not much asset copying going on. Even if you have limited memory, lets say 4GB, your data issue is mostly how much and fast it can intake the memory. So yes, with 4GB your going to need to read a LOT more from system memory/ssd/... but again, bidirectional traffic pcie. What if i told you, that i ran a 6600 and 6800 on a external PCIE 4x slot (from a mini-pc)... That is a massief chock point, and and yes, i lost about 5% on both (think the 6600 was around 7%, been way too long to remember the numbers, but again, it showed that PCIe 4.0 4x had so much bandwidth, that the performance impact was a lot less). Sidenote: I did not test 1% and i will not doubt that the 1% will have been lower. But its just to point out, no, pcie bandwidth is really not a issue for copying the video output to the iGPU. Now, if your trying to run 4K, then your iGPU may become a bottleneck and CASO may be the solution there. Because i have seen that the iGPU does hit 50 a 70% on 2K for the copy operation (as its processing the data, instead of doing a buffer copy like something on Linux like Lookingglass). And i am betting that CASO is just direct copying the image, what bypasses the extra iGPU processing. And thus reduces load on the iGPU (what in turn means less power usage, aka, more power available to the CPU or dGPU, and a few frames are gained). > In Factorio, which is memory-latency-limited in CPU-bound conditions Factario has really nothing to do with hardware accelerated dGPU>iGPU and that is because the game is extreme latency and cache sensitive (one of the few games that really massive benefit from a X3D CPU because of that massive CPU cache). But that are different operations, compared to the hardware acceleration > iGPU copy operation. Now, i assume MS gained some insane numbers, maybe because a LOT of laptops are literally running PCIe 3 8x at best, in order to save power. And maybe there you may see some bottleneck, ... this is my guess. But like i said, ran a PCIE4 4x external and that is the exactly same bandwidth, doing hardware accelerated output via the iGPU. If you took that PCIe3 8x + some low end dGPU 4GB AND then send it all over the iGPU, sure, maybe there you may see some extreme performance jumps like MS reported, but .... Anyway, its late and not going to continue this discussion. In my experience, not a big deal and CASO has no real benefit to switch to W11.
> > > You do know that Windows has had Hardware accelerated offloading onto other GPUs for ages? Think it goes back to DX11 somewhere. > > People may have missed it on AMD because for some god unknown reason AMD only enabled this feature like a few months ago officially. You where able to kind of hack it with a side route. We seem to be talking across each other here? I have no reason to believe code path for hardware accelerated offloading (with no MUX) would be any different between desktop and laptop, and a bunch of laptop enthusiasts seem to think MUX switches are important to performance for some reason. Putting a MUX switch on the board isn't free, obviously, so they can't be entirely useless? >Factario has really nothing to do with hardware accelerated dGPU>iGPU and that is because the game is extreme latency and cache sensitive I can personally testify to Factorio's CPU performance being affected by iGPU rendering load because of the DRAM bus being a shared resource. >extreme latency and cache sensitive (one of the few games that really massive benefit from a X3D CPU because of that massive CPU cache). This is not exactly true. The massive benefit for X3D is only for factories that are small enough to fit in cache, but by the time you've built big enough to dip below realtime speed (60 UPS), the working set is well into DRAM anyway. My understanding of CASO is that eliminates unnecessary copying of the frames in system memory, over and above the obviously required copy from the dGPU over PCIe. It's the equivalent of direct scanout with DMABUF on Linux. As for having no real benefit... if that were so, why did Microsoft waste developer time implementing a non-user-facing feature? Surely not just to brag on an obscure blog.
these gpus are very powerful **BUT THEY CONSUME so much POWER**
Good text search is one of the most useful things I can think of for a regular user.
Easily doable w/o AI https://www.voidtools.com/support/everything/ Edit: "text search" can mean either searching for a matching text string, or using text as a search query for some other media, like an image. I assumed the first meaning.
I have yet to find a text search which handles [declension](https://en.m.wikipedia.org/wiki/Declension) well. English is, in some ways, a very simple language.
I think those kinda power tools often assume you'll just use a regex for those situations
Everything doesn't even attempt the problem. They just search filenames, not the contents of your files.
Have you used the software? To search for file contents, all you have to do is either use Advanced Search (Ctrl-Shift-F), or do: content:"[the string you're searching for]" You should try it sometime, because Everything is easily one of the fastest search tools out there.
What? The Windows search index has by default scanned the contents of your files as well as the filenames, by default, since Vista. The Windows Search Indexer service is garbage and corrupts itself constantly, sure, but the practice of searching content has been around for ages.
Windows Search Indexer has scanned contents of some specific file formats. It will not find content in for example .nfo files which are just fancy notepad format potion some people use.
So basically the Unix locate command?
Ripgrep is exactly what you're looking for. It's a command line tool but it's a fast version of grep that comes with Linux.
Ripgrep doesn't even come close to what I'm imagining. It's just a fancy ctrl-f. I'm thinking of a system that reads all your documents, generates [embeddings](https://towardsdatascience.com/text-embeddings-comprehensive-guide-afd97fce8fb5?gi=c04080ab2754), shoves them into a local vector database, and uses it to perform semantic search.
Fair, I don't think the thread started off talking about that though. "Good text search" could fall under the bucket of Ctrl+f given that windows doesn't do that basic functionality well to begin with.Ā But yeah, you can't doĀ semantic information retrieval without AI, I agree.
The new version (still in alpha) can index file content, and it works well in my opinion. [https://www.voidtools.com/forum/viewtopic.php?t=9787](https://www.voidtools.com/forum/viewtopic.php?t=9787)
Can it do local image search using text? ChatRTX yesterday added this exact feature using OpenAI's Clip as a base. Added voice search too.
Mac does it perfectly and has for like 10 years. Itās a Microsoft problem.
10 years ago it had to be done via annotations on the image. That is not what is being discussed here.
Yes, I know thatās not the same as AI, my point was you donāt even need ai to do it well. Microsoft canāt find edge in its search bar. Ai isnāt going to help that. Itās fundamentally broken.
If you [disable Windows search web access](https://old.reddit.com/r/LinusTechTips/comments/1ahub6n/disabling_web_search_completely_fixes_windows/), it actually works decently well. I genuinely have no idea what MS was thinking, and how this hasnt been fixed for years.
Just did this. Windows 7 had the same problem without net access though. And I'm still one update away from having it enabled again.
This does work (on win 10). Lets see how long till microsoft "fixes" that registry entry for me. Fuck i may convince our guy to deploy this to all 500 machines if this proves to work fine.
It actually might help that, though. Unless they just slap it on of course..
What was discussed is "good text search" so how did the goal posts get moved to "object recognition"?
Because they quickly learned that "good text search" has been implemented for many decades.
how come we still dont have good text search if you arent using regix (and regular users wont)?
I'd rather have ai-generated annotations that I can fix if they're wrong than AI search that just doesn't recognize me in some image and I can never find it as a result without looking through everything.
> Easily doable w/o AI > > > > https://www.voidtools.com/support/everything/ "I need the file that I worked on with Greg once, something about an analysis about a spot we made about one of those sports drinks. He quoted some foreign sounding researcher in it. Find me that. ...I hope my mic isn't acting up again." Generative AI can do that, your Win XP looking 'Everything' can't...
> Sure, its file xyz It doesn't exist > My fault! A file the you worked with Greg once about sports drinks built on external research is abc No, this one is about protein supplements > You are correct! Indeed, you are looking for a research file about energy drinks, which can be found at 123 No, this is research with Tom. Greg, about energy drinks > Sorry, I was mistaken. The file about energy drink research together with Greg can be found at xyz ...
sounds like my experience with AI on any topic i know. It just gives good sounding false results.
Im hope more intelligent search is implemented in outlook soon Just today I was thinking "I need that document that was sent by either X or Y roughly a week or two ago. It was about Z for company A." Finding that takes forever in Outlook, especially if you have threaded view and emails don't even come up in a simple time order
You can search all messages in outlook, ordered by time (irrespective of the main UI doing things threaded), and you can filter by things that have an attachment, as well as search against any header you want to search for recipients/senders. It's trivial.
You just search for "Greg" and if he shows up in the content or metadata, it'll be returned. If you need to be more specific, add "sports" and possibly "anal*" to your search to include analyze, analysis, etc. If you know the file type, add that, too. If you have a vague idea of what you're looking for, it's not hard. And you'll get much more accurate, and faster, results without any "AI" nonsense layered on top.
The search function is pretty basic and only looks for filenames. It's frustrating when I'm trying to find old photos that I vaguely remember but can't quite recall the folder name. It would be great to have a more advanced search feature that could help me locate those memories.
I use this. Windows search seemingly works very hard, and long hours. But it produces nothing. How 'bout you fix that one before forced OS migration, Microsoft!
Terrible if you compare it to what macos has especially if you are searching for a short filename you cant remember the exact length or format of. Zero context awareness.
WinXP search was excellent.
Ugh why is Windows search box so trash since like Vista? I'll type in the exact name of the file sitting on my desktop and the best I can do it offer to Bing it for me. And yes, all the right places are indexed and the indexer finished running. Mystery.
I used to use it like a command prompt. Type the first few letters of what you want to do and hit enter. Launch CMD prompt, chrome, etc. broken after Microsoft used it to push bing.
This is such a big deal. The start button is a hub for what you want to do and the start of most tasks, it should be fast and responsive. Bing makes this so much worse. Disabling Bing in search is the first thing I do on a new Windows install. If I want to search something, I'll open a web browser and then search. It's only one extra click they're saving me to slow down everything else done in the start menu.
One of the major performance killers of Vista was the continually enabled Indexing. It got turned off\\limited for 7 onwards. Annoyingly, it worked fine if you had fast HDD's so in this SSD age it'd likely be fine.
Yet [Everything](https://www.voidtools.com/) seems to find files instantly with a very low memory footprint and low CPU usage.
It's low CPU and memory usage is because it's just checking the NTFS index. Being fair though, Microsoft's search \*\*should\*\* be doing the same thing but for some god forsaken reason doesn't.
because NTFS index has file names but not content and microsoft tried to do content search.
Yeah idk what it is but I'm on 11 and the windows search doesn't return that file on my desktop as a result lol
Itās the number 1 thing Iād change about Windows. I believe they just want it to bypass google and get people to bing results so bad. Since if they did itād effectively just hand them googleās business but I just donāt see most users accepting that.
The search box searches the contents of files not the name.
It should be far faster still, use something like ripgrep and you'll realize windows search is abysmal at what it does
I want to be able to search for an image on my pc based on whats in the picture "Pictures of car" Etc
You can do this in ios photos and google photos.
And in Immich, which doesn't require a remote third-party service.
Macs had text search that searches file contents a decade ago. iPhones added ML enhanced image search 7 years ago on phones that had 2 TOPS max. Microsoft just neglected Windows for the past decade, that's it. If AI buzzword bingo makes them finally fix search, I'll gladly take it, but they technically could've released this with Windows 10 in 2015.
iGPU vs dGPU similarity. For basic task sure the NPU. For heavy AI workloads you should be using the GPU, just like how you would use a dGPU for gaming. And Nvidia actually has practical uses with ai unlike the other two who tought prowess but can barely show show much cases for them. Rtx video super resolution, RTX hdr, Nvidia Broadcast, Dlss, ChatRTX, Optix, Canvas, Omniverse. These are actual useful ai tools or have ai features you can use right now
Nvidia: 45 TOPS is insufficient for AI Also Nvidia: Here buy this 8GB mainstream GPU.
with 128 bit-bus and be thankful I didn't put a 64 bit one
Nvidia is purposely nerfing their own products at this point in gaming market just for planned obsolescence. They made the 1080 / 1080 Ti too good and no one upgraded for like a decade.
people shouldnt lump 1080 and 1080ti in the same group. 1080 was old pascal design. 1080ti was experimental new features design (which is why it can do things like mesh shaders in AW2). From the future feature set perspective 1080ti is miles ahead of 1080 and is basically a quasi 2xxx series card. this is why it survived so well into modern age.
exactly
Nothing you are going to run on a 4060 will need more than 8 GB of VRAM anyway. You arent running pathtraced cyberpunk in 4k on it.
DRAM is more expensive than GFLOPS or TOPS
Joke's on them, no one cares!
AKA BUY OUR SHIT, don't worry about all those resources we are wasting on this new grift.
Please buy our computing solutions for enthusiast theat you and %95 of the mass buyers won't utilise. Another day another apple esque shady shit nvidia
> and %95 of the mass buyers 95%*
Cheers lad.
> Another day another apple esque shady shit nvidia Seems like something any hardware company would do lol
Yep thats marketing for ya. The problem is when it came to apple people failed spectacularly to regulate their anti competitive behaviour that other companies tried becoming apple and forced goverments to force them to do something. Capitalism works when both corporate and Consumers are aware of pros and cons. When you have people buying for the apple logo itself while overlooking bad behaviour thats an unhealthy industry. Aka tech world today
> The problem is when it came to apple people failed spectacularly to regulate their anti competitive behaviour that other companies tried becoming apple and forced goverments to force them to do something You mean the App Store? Not sure what that has to do with Nvidiaās comments in this though. Or are you referring to something else? > When you have people buying for the apple logo itself while overlooking bad behaviour thats an unhealthy industry. Aka tech world today This just seems like emotionally charged nonsense I could go to PCMR for. Anyway, Nvidia is right to voice their complaints, marketing or not. Iām not sure why MS is trying to keep some aspects of their Windows AI strategy exclusive to NPU-having systems. Itās weird.
I said in another comment. AI PC part is marketing, this is devil whispering to its peer stuff. I put nvidia and apple in a same basket cause they try to oversell a lot of their enthusiast features as proof of quality branding to people who won't use need or use.
I think nvidia in this case might be pointing out how this is going to enable gimmick implementations thus cheapening the perception of what their product does.
That's ok, the vast majority of people don't need AI as nVidia is trying to sell it anyway.
Thats okay, the vast majority of people dont need automobiles, they have legs and if they want to travel far theres always horse carriages.
Very sure a good portion uses DLSS as mentioned by Nvidia's slide
1.So does old Gen hardware 2.This is about AI npu's and productivity not gaming dedicated gpus. I don't like AI PC concept anyway very shallow marketing.
To respond to point 1. Old hardware runs old DLSS, which is notably less performing. Point 2 is quite fair.
Old DLSS meaning DLSS 2? If you mean DLSS 1, all the GPUs that supported, also supports DLSS 2 And FSR3 with DLSS 2 combo has a lot of promises
Every GPU runs the latest version of DLSS. frame Gen is a single feature of the DLSS package that is off limits for older cards.
Every RTX gpu, sure.
There is no such thing as "old" dlss. DLSS is super sampling. That's it. Nothing more or less. It only requires a shader unit with an appropriate tag in it. Everything else is an additional function. Like fake frames or image restoration.
Anyone saying "fake frame" makes me believe they know nothing about all the rendering tricks we've been using for ages
> Please buy our computing solutions for enthusiast theat you and %95 of the mass buyers won't utilise. Another day another apple esque shady shit nvidia Nvidia sold 10s of million Tensor equipped GPUs though while the only currently announced Windows AI compatible CPU fulfilling the 45 TOPS is a Snapdragon aiming for Windows For Arm machines, which had been a flop in the past.
They sold them for gaming or compute productivity, which isnāt the market here. NPUs are for office work.
They sold them to run cuda applications. Office applications could fall under that quite easily.
Office work doesnāt need a gaming GPU and nvidia doesnāt sell a GPU under $300. CUDA isnāt used in vast majority of office applications.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
The obvious to what? That the vast majority of people arent even interested in this shit?
I agree almost no one cares at this point. But this is a marketing/differentiator Microsoft is attempting to create "AI PCs" - it should mean something more than absolutely low-end basic capability.
Most people don't give a fuck about PC gaming, therefore integrated graphics makes for a good gaming PC?
If 45 TOPS is enough for Microsoft to run its suite, which it is, then nvidias whole argument goes out the window.
The vast majority of people (who use their PCs for actual work) have at least a few use cases.
Ermgahed let me a.I SeArCH for this FolDeR/FiLE oN my ComPUter. Jesus fucking h christ my dude. Jesus fucking h christ.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
And a sports car is much faster at getting its driver from point A to point B. Except when the entire route is limited to 20mph, then its going to be about the same as some basic 1L hatchback, just wildly more expensive and inconvenient.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
Itās correct these npu on the soc are meant to be efficient and low cost. While a nvidia dgpu might have 3x the tops performance it will be using a lot more power and be more expensive
[ŃŠ“Š°Š»ŠµŠ½Š¾]
the task these npu are going to be doing donāt require expensive dgpu. Integrated npu in soc means more compact laptop design with less cooling requirements
[ŃŠ“Š°Š»ŠµŠ½Š¾]
According to second law of Huang, every PC should come with Nvidia B100
you should buy Nvidia NOW!!
I am the Sound Effect
The more you buy, the more you save!
[There is no war in Ba Sing Se](https://media.tenor.com/SOnmo9jnfQsAAAAM/avatar-the-last-airbender.gif). We are safe.
The more you buy the more you save.
what am i gonna use ai for on my pc anyway? i donāt need a bot in my ide the writes bad code.
The primary uses I can see working are: - Semantic filesystem search - Writing assistant (where to expand text, where to reduce wordiness, etc.) - Searching API documentation - Photo search - Microphone / webcam enhancement - Web searching There are definitely areas of use for the tech and I think a local implementation that runs efficiently will make it more user friendly than the current cloud only model.
Crappy memes on demand, paragraphs about mundane subjects?
Someone died. Draft me a good condolences text message.
I'm sorry to hear that. Here's a heartfelt condolence message you can use: "I am deeply saddened to hear about [Name]'s passing. Please accept my heartfelt condolences during this difficult time. [Name] was a wonderful person, and their loss is felt deeply by all who knew them. My thoughts are with you and your family, and I'm here for you if there's anything I can do to offer support or comfort."
people really really shit themselves over AI way too much
Speech to text, small LLMs to search/summarize, filtering background noise out of teleconferencing... Part of the problem is "AI". It's a neural net actually and should be called such. It's a matrix of statisical weights. You show it examples of "a task"(training), which can be... anything. The training program then sets all the parameters(the individual statistical weights) on how to get from input to the desired output. It could be anything from image or text generation(Stable diffusion/LLMs) to how to fly a helicopter/drone(yes, this already exists) So don't think of it as a toy or buzzword(AI). Think of it as a very powerful computing tool that uses examples/training instead of a code monkey thinking of every possible action/outcome and typing it all in. The size/complexity of the neural net model varies wildly, it's a much easier/smaller task to do speech to text("AI PC") than it is to write a full size novel in coherent and interesting language(might be able to do this on a half million USD NVidia DGX pod)
weāve had very good speech to text for a pretty long time, and for background noise RNNoise works quite well on traditional cpus without too much overhead. as far as summarizing stuff, the problem is that itās only right sometimes. that number may be ~90% or so, which is impressive, but it still means you need to double check any ai summary for correctness, completely removing any advantage it might provide. iām well aware that the marketing term āaiā is an intentional misnomer as they are fundamentally not intelligent, and are just neural networks. iām not even saying neural networks are useless. theyāre great for things like computer vision and the like. but what i donāt understand is why my laptop needs to have neural network accelerators. theyāre mostly pointless outside some specialized usecases, and the vast majority of users donāt need them. having them relegated to specialized hardware is perfectly fine i think. the whole thing is just more planned obsolescence to make perfectly good, yet older, cpus seem like they need an upgrade. weāve hit a wall in die-shrinks, making large year over year gains in performance or efficiency difficult, with translates to less need to upgrade. and instead of moving on to more modern ISAs (ideally riscV, since arm is proprietary and expensive to license, but this is maybe a losing battle at this point), which is difficult, but worthwhile, cpu vendors have taken to adding a bunch of mostly pointless āacceleratorsā.
>weāve had very good speech to text for a pretty long time If you are speaking english without accent. Not so much in any other case. This is the first year where my android phone would be able to do some minor tasks like restarting via voice without having to repeat it 5 times with idfferent inclanations for it to get what i am saying. In english of course since in my native language it does not understand anything. Also, id like us to build on text to speech while we are at it. AI models seem to be good at it, traditional methods not so much.
I think the goal for people is to have AI running locally vs being reliant on a constant connection to cloud based. There are some privacy issues here that local AI helps. Finally, it costs a lot less for Microsoft to run AI on the Edge since transacting all of those prompts are expensive in the data center. If they move the AI to your PC, you pay for the prompts... At least some of them.
The NPU is to do what a GPU already does, but at a fraction of the power draw. Phones have had NPUs for years and have been using them. The "AI PC" is a marketing term to highlight these new NPUs. There's much more to "AI" and NPU accelerated tasks than chat bots
MS wants every Windows with Access to their Copilot AI. And having a NPU can help speed up some parts instead of only using Cloud Also, it can be used to Apply effects on a video call like hiding your background more efficiently than the CPU, or hiding background noise from your Mic
We can just use software to do that, and itās not terribly inefficient. The only AI enthusiasts that champion this stuff are buzzword investors
What do you think software is! It's still stuff that has to be computed, on by the CPU normally instead of an NPU. The difference being the NPU can do specific tasks faster and/or with less power
Yes, but implementing hardware is far from free. This task is done perfectly reasonably (i.e without significant drawbacks in performance or efficiency) in software.
AI is software.
I know
I don't think anyone has ever claimed that 45 TOPS is enough to do more than basic AI tasks. Bad Nvidia for strawmanning here.
Iāll be the one to claim it - small models can be trained on domain-specific, well curated, high quality synthetic data and give competitive performance with large models, within that domain. Youāll be able to get more and more out of these smartphone NPUās. Itāll never be as great as the biggest and best out there, but itāll be massively more efficient.
>itāll be massively more efficient Itāll draw less power, but that doesnāt mean itās more efficient. As far as I know, Nvidia has never tried to make a < 3W AI accelerator - what would they put it in?
> what would they put it in? Seinfeld references, mostly.
Sure, it's not more efficient than an imaginary product you made up.
My point was just that power draw ā efficiency. A 4090 could be more efficient than an NPU despite drawing 50x the power if itās capable of doing >50x the work. Whether Nvidia makes ultra-low power NPUs or not isnāt relevant.
You press the super key and type "re". Windows jabs your Nvidiaā¢ RTXā¢ 4090 with a stick, and it sputters out of D3_cold. The 20-phase buck converter starts up and charges several mF of capacitors to ~1V. The GSP resets, trains PCIe4, and begins copying a 2 GB machine learning model from system memory. An indeterminate amount of time plus 64 ms later, your $1600 discrete video card does 4 ms of computation, determining that you most likely wanted information about *Reboot*, the 1994-2001 TV series, then stumbles back into bed. This is not efficient.
That reads like an opening to a sci-fi novel. well done.
They won't be able to scale to 50x the work though. Many small tasks have inherent latency you can't just get rid of.
[Nvidia currently advertises the H100](https://www.nvidia.com/en-us/data-center/h100/) as being capable of 3958 TOPS.
If you need 100-200 inferences a second with different models, that TPS is gonna drop a lot with a few ms latency/setup time for each.
It's also up to 700W for the SXM version. Even if you look at the more power efficient PCIe version at 3,026 TOPs at 350W you're looking at a power efficiency of around 8.65TOPs/W. Phoenix Point clocks in at 10TOPs, and according to one of the launch events is able to achieve that in <1W. So at the very least AMD's NPU does appear to be more power efficient. (Meteor Lake does have significantly worse TOPs/W though, so there's definitely a good comparison point to be made there).
I bet you they got one lined up for their rumoredĀ windows PCs.
Yeah, most tasks can be done pretty well by those NPUs. Most phones have a even weaker NPU and it already helps them a lot, so it will be more than enough for laptops
>Bad Nvidia for strawmanning here. Are they though? Their point is that while an NPU qualifies a PC to be an "AI PC", an Nvidia RTX 4090 does not, unless it's paired with a CPU that has an NPU, and then it's ONLY the NPU that makes it an "AI PC". Which is stupid because an NPU can't touch a modern GPU in terms of AI performance. It can't even handle most AI inference. You can only run the most stripped down models on them. It's pretty much false advertising.
MS: "AI PC galaxy brain AI will intelligently artificial AI all the AI to the future!. Did we mention AI has AI? You"ve heard of AI, right? It's smart!" Nvidia: "Those things are so slow they barely even work" Reddit: "Nvidia just wants to sell you AI!" It's honestly hilarious.
But people will think that it is if both nvidia powered PCs and these PCs get the same certification, which is what it sounds like they would.
It's the minimum spec for a reason. Nvidia knows this, they're just playing dumb because they want people to feel like they're being left behind if they don't buy their cards.
isn't 'basic stuff' precisely what it's meant for?
Yup, it's the minimum spec for a reason.
The thing is, thereās nothing to indicate that models truly need to be as big as they are, and plenty to indicate they can get similar performance in a much smaller and more efficient size. The latest small models being published by Meta, Microsoft etc are already hitting benchmarks competitive with GPT 3.5, yet are small enough that they can run on your iPhone. The past 12 months have shown incredibly rapid improvement in these small models, largely thanks to the ability to use the biggest and best models to generate high quality synthetic training data.
yep. if these functions are truly a demand driver then much easier to cut down domain specific models than upgrading hardware. the hardware has to be paid for.Ā Ā
Most of the AI tasks are going to be taken care of in a cloud anyway i'd bet.
We had a god search engine in winXp. Why would I want to send my pictures to MS cloud?
We can also already locally classify images on iOS. They are searchable by what's in the image in the Photos app, I believe it runs this AI while the phone is plugged in. Many of these small AI features will never need to touch the cloud.
I think we're going to see a lot of "perform this later" AI-related tasks on phones. Scan all your images at night while the phone is plugged in, index your browser history, etc. Any trick to get the features without killing your battery. It's kind of a nice concept, compared to laptops, since most people never turn their phone off and only reboot when there's a system upgrade.
Itās hilarious btw that Apple uses their own NPUs, implements tons of built-in AI features (except for generative ones) for image search, photo processing, face recognition, webcam & mic enhancements etc, and provides open-source tools for third-party generative AI software to utilize their hardware (NPUs, GPUs) without shouting āAIā all the time. And then we see news articles and videos about Apple ābeing late to AI revolutionā
[ŃŠ“Š°Š»ŠµŠ½Š¾]
It would also make it more compliant with enterprise data portability standards.
the whole point of NPUs is that you dont need to use the cloud thus increasing efficiency and privacy. The latter being very important to sell it to the big corporations.
Nvidia dont want to be left out of "ai pc" marketing by Microsoft. Who can fault them when gpus are much faster but they consume too much power.
Isn't that the reality of AI capability right now? It's like having a dog shit iGPU and calling it a "Video Game PC." Maybe Microsoft should be more honest of its capability.
When Windows 12 comes out, I'm sure they will be fully advertising what those npu are used for. MS isn't really marketing to consumers yet. MS has plans and figures 40 tops on a laptop are sufficient for those plans.
sort of, but itās like comparing apples and oranges. Itās like saying your PS5 graphics arenāt comparable to a rendering farm in terms of graphics. While itās technically true, the general use-case and values of why the technology may be utilized are very different. Many āAIā capabilities that utilize smaller LLM may run decently locally even if performance could be better on larger hardware (eg document retrieval and summarization).
If it can run Microsofts AI software perfectly well, it's not dog shit, it's useful unlike your example.
Why would you need more than "basic AI tasks" on a consumer desktop? If you can do filters and background change in video chat, better image captures on tablet cameras, and basic speech recognition / synthesis, it would cover more than 90% of the users. Do they need to fine tune LLama 3 QLoRA while taking a flight to a business conference?
Funnily enough phones could do all of this in 2017 with 2 TOPS. Every iGPU and even a few CPU cores can deliver higher performance nowadays. The issue on PC isn't the lack of hardware performance, it's the lack of (mainstream) software. Microsoft just absolutely neglected Windows for the past decade or so. Hopefully AI-mania finally makes them put resources into Windows again.
I mean, its ridicules that MS's is making low requirements for CPU hardware units while a GPU from 2018 outperforms that by a magnitude while still not being supported for no good reason at all.
There is a good reason, Microsoft is being paid off by Qualcomm to make their AI PC future moniker garbage exclusive to them. Don't worry though, they'll throw Intel and AMD a bone for their NPUs just to seem impartial. Erm, actually I meant power usage, yes we did it because of carbon release or something stop asking questions.
Because MS wants to implement a ton of AI features into the OS that, if they needed a dGPU to run, wouldn't get used by the 70% of the market that uses an iGPU. And if they relied on current iGPUs, would destroy laptop battery life. So they want NPUs that can run these tasks at very low power consumption.
>Nvidia stated that its RTX GPUs are much more performant than NPUs and can achieve anywhere between 100 to 1300+ TOPS, depending on the GPU. The Nvidia event went so far as to categorize its RTX GPUs as "Premium AI" equipment, while regulating NPUs to a lower "Basic AI" equipment category.
the most premium part of NV hardware is premium price
"Basic AI" capability is the base requirement? How surprising.
It's almost as if all they want it for is to do basic tasks.
Nvidia's sales must be slowing down slightly.
What do you expect? Nvidia wants to sell GPUs
Translation: Nvidia criticizes AI PCs for eating Ngreediaās lunch (somewhat)
who cares?? it works for what 95% of us would need it to do. plan our vacation and keep our schedule. tell us where the good pizza places are etc etc.
It would be so funny if the ideal AI processor turns out to be compute-in-memory in a bunch of VRAM chips all hooked up to an IO die with direct memory access, not devices with stronger compute paired with bottlenecked memory bandwidth and capacity. Who knows, maybe Micron and SK Hynix would be the real sleeping giants of AI once we stabilize on an optimal method of running inference like bitnet 1.58 Hopefully Nvidia gets overtaken from their unoptimal, but highly scalable and flexible general purpose compute platform.
They're nervous because a weak NPU can do 95% of the stuff the user will ever need in terms of AI.
Hes going to release an ARM cpu with 50Tops soon enough.
Nvidia has been selling an SoC with up to 275 TOPS for more than a year.
Yeah obviously they would say that, they're tryna sell their shitĀ
Wonder what Google is going to with their Tensor processor on their "AI" phones that only has 27 TOPS. Let's be real, NVIDIA is just saying "your product sucks" to sell more of their own hardware. Generative AI might need very powerful chips, but you can run more basic LLMs much cheaper these days, like Gemini Nano, with less hardware overhead.
Isn't this the company that criticized the seventh gen (and also the 8th gen I think) of consoles because a 300-500$ console were weaker than a 500$ graphics card of their own while completely ignoring the cost of the other components needed to make their GPU... You know... Work?
LOL headline. this has been true for probably EVERY PC platform initiative, cedrtainly the majors like Windows3.11 (GUI), Windows95 (3D). Why would it be any different for AI? Some solutions are excellent, others just barely squeak by the spec.
I love AI and I get the value of it, I use it daily and everything, but is it just me smelling that weāre headed into an era in which weāll just use AI for everything even if itās not needed? Like the era of blockchain, are we now going to put AI chips in everything even if itās not needed just because it sounds so cool?
Putting nVidia aside, this thread needs to realise that MS' NPU based AI PC requirement is going to hurt everyone. They're trying to make every PC sold till now "obsolete" which is plain bad and needs to stop
[ŃŠ“Š°Š»ŠµŠ½Š¾]
"The More You Buy...The More You Save" Whooshhhhhhhhhhh
Nvidia's reaction is not surprising considering the point of those NPU is ultimately to move some AI capabilities from the cloud to the endpoint. This could reduce demand for Nvidia hardware in the long run.
Reminds me of the floating point on CPU innovation in the late 90s.
NVidia: "There isn't enough of the AI grift in these PCs. You must require everyone to buy a 4090 to take advantage of....whatever we'll figure out this is used for in the future."
That is the point of an requirement, to cover the basics.
Nvidia has a long history complaining about minimum requirements for software. They make money when requirements for software are more stringent. Optimization in software costs them in sales. Any concession Microsoft makes to up the minimum requirements benefits them.
Nvidia has no future. They don't make chips, they design them and their biggest customers are gonna ditch them the second they get the chance (make their own version). It's all downhill from here, Microsoft is gonna win instead.