In addition to Recall, Microsoft is working on a new Windows Search based on a semantic index that could address the problems with the current search index. Windows 11’s new search engine could use AI to power your local search results and even look through audio or video files.
Windows 11’s Recall feature was one of the flagship highlights of Microsoft Copilot+ PCs, but things didn’t go how the company hoped. After outrage due to privacy and security issues, Microsoft pulled Windows Recall and promised to bring it back in October after we reached out to Microsoft.
While you can search your past activities with Recall, you could soon even sift through video and audio files using AI in Windows 11.
Microsoft researcher Xeno recently spotted “Intelligent media search” references in Windows 11 Build 27695. It contains several feature references that point towards an AI feature that can search for content inside the video and audio files.
This AI magic will work after downloading the relevant models, which will make this feat possible. It appears that the feature will first transcribe every video/audio file on your PC. After that, you can perform a content-based search.
Windows Latest expects that you can search for specific words said in a conversation. For example, you could search for details like a project name or important statistic and see relevant videos/audio files appear in that result.
It could even go beyond that and present the exact timestamp when the word appears in the file or emphasize its relevance in the conversation.
Intelligent Media search could be related to semantic search
It’s possible the intelligent media search could be related to the Semantic search that Microsoft announced at the Copilot+ event.
As noticed by Windows Latest, Microsoft pointed out in a recent blog post that “Recall AI is powered by Windows Semantic Index.”
The benefit of this index is that you can interact with it using natural language. Soon, developers can access Vector Embeddings and RAG API in the Windows Copilot Library.
Recall AI’s early demos showed that it would use snapshots to help you find anything you previously did on your PC. The “Intelligent media search” would be a step above that. We like the idea, but like Recall AI, this will also be exclusive to the Copilot+ PCs, which debuted in June 2024.
Copilot+ PCs have the necessary hardware to run AI models locally with 40 TOPS and above NPUs. We don’t expect the feature to arrive on the regular Windows 11 PCs because they lack the necessary hardware.
Dev-party integration in Windows 11 AI
Microsoft also wants to extend Natural Language search capabilities to third-party app developers.
Devs can offer a similar natural language search experience in their apps. Think of a finance app where you can search for a past action using natural language. The search response will be faster since all the data will be present on the PC.
Recall AI is making a comeback in October 2024 for testers after a four-month hiatus. You can remove it via the Control Panel’s Optional Features dialog box if you don’t need it.
The post Microsoft could redefine Windows 11 Search with AI to search through audio and video files appeared first on Windows Latest