Virtual Assistant F...
 
Notifications
Clear all

Virtual Assistant Forensics

10 Posts
5 Users
0 Likes
867 Views
RolfGutmann
(@rolfgutmann)
Posts: 1185
Noble Member
Topic starter
 

With the new Google smartphones Pixel and Pixel XL the Goggle Assistant - a personal AI-based learning Virtual Assistant (VA) joins the group of Siri (Apple), Cortana (Microsoft) and Alexa (Amazon). There was Google now, but unclear to me if still standalone or transmerged into Assistant.

Most VA reside in the respective cloud. Some are hybrid local-cloud. Their deep personal intelligence of the user is not imaginable and cannot be measured exactly. But for sure they have a high level of highly accurate context-based (time, location, emotional grade, environment) intelligence which could help to find evidence either on the suspect or as a source of social-environmental evidence (running mics in a multi-person affective crime site).

But how to get this evidence out and accessible? As BigData will not be handed over by a court order and on the cloud provider's distributed storage systems geo-decentralized and affected by many legal borders like e.g. Privacy Shield (EU) you will not get the 'real' data. Where the real physical-virtual location of the data will be determined is not solvable as the raw data is not the aggregated deep-learning AI-based intelligence located somewhere else maybe.

How can we get evidence out of VAs?

Their intelligence is unbeatable but out of range. This should change.

 
Posted : 07/10/2016 10:02 pm
jpickens
(@jpickens)
Posts: 130
Estimable Member
 

If your google account is default, then your ALLO searches, and any other related history is all stored in your Google profile/cloud. Just tested this on myself to verify. I'm sure this could be disabled, but for forensic investigations it may not be that way for all persons.

I have not tested this but I would also assume some DUO and ALLO data would be stored locally to the device.

 
Posted : 07/10/2016 11:37 pm
RolfGutmann
(@rolfgutmann)
Posts: 1185
Noble Member
Topic starter
 

Good point!

We wait for getting from Germany the Google Pixel with Android 7.0.1 (Nougat) to test where the Google Assistant stores requests and response. Google mentions that the traffic will be encrypted, but the question is how the AI architecture BEHIND runs. Intelligence based on BigData first runs on the OTT-player's server. Its questionable how offline AI-intelligence will succeed as in general the big potential lies on OTT.

 
Posted : 08/10/2016 11:58 am
RolfGutmann
(@rolfgutmann)
Posts: 1185
Noble Member
Topic starter
 

iOS 10 brings Sirikit to developers for app integration for a limited range of functions (but also payment!), see here the slides

http//devstreaming.apple.com/videos/wwdc/2016/225e97ww3majsu3bvt8/225/225_extending_your_apps_with_sirikit.pdf

 
Posted : 08/10/2016 12:47 pm
UnallocatedClusters
(@unallocatedclusters)
Posts: 577
Honorable Member
 

I recommend searching for MP3 and other audio files on the phone and listening to audio files that appear in folders unrelated to locations music files the phone owner would normally save, or have saved audio files to.

I imaged my personal Android phone (LG G3) to perform basic research and discovered that Google Maps was storing the verbal directions spoken by the Google maps voice as MP3 files in a system folder.

So when Google Maps is providing verbal directions on an Android phone such as "Turn left at the next intersection", those verbal directions are being stored as MP3 audio files on the Android phone in a system folder.

My thought is that (and I have NOT confirmed this) AI software such as SIRI or OK Google is also storing the "speech" we hear SIRI and OK Google speak as MP3 audio files on the phone itself.

One probably needs root access to a phone in order to extract all system folders containing such audio files, but listening to these audio files could enable one to reconstruct what the AI software was saying to the phone owner.

Hope this makes sense.

 
Posted : 12/10/2016 1:12 am
RolfGutmann
(@rolfgutmann)
Posts: 1185
Noble Member
Topic starter
 

Makes totally sense from my point of view. As Siri and the Google Assistant combine speech and text (displayed in the UI) there must also be a text based evidence to find somewhere. Within iOS code the conversations between Sirikit and the user input by voice mostly should be traceable by Blacklight e.g.

Who did Siri conversations already image out with Blacklight?

 
Posted : 12/10/2016 1:36 am
RolfGutmann
(@rolfgutmann)
Posts: 1185
Noble Member
Topic starter
 

Google has three products running as fully or partial Virtual Assistants Google Now, Google Allo and Google Assistant. Are these just different frontend GUIs but behind the same AI- and DNN-based server-clusters? How does flow information between these three entities but linked to the same gmail-account?

As e.g. Google Assistant evolves very specific to several compared gmail-users as content- and interaction-related machine-human-relationship is getting built, how exactly can Law Enforcement know what intelligence is residing on Google? There is a certain risk to missing evidence on this layer.

Has anybody run a large investigation with crucial/central Virtual Assistant evidence found to solve the case? Our know-how in this field is weak but looming and we want to prepare internal training for all Virtual Assistant on the market including newborn Bixby.

Please share your experience with us. Thank you.

 
Posted : 04/05/2017 9:37 pm
SamBrown
(@sambrown)
Posts: 97
Trusted Member
 

RolfGutman, as always there are a lot of questions and no answers. But it is always entertaining to read your threads. lol

 
Posted : 05/05/2017 6:27 pm
RolfGutmann
(@rolfgutmann)
Posts: 1185
Noble Member
Topic starter
 

There is a study out about Personal Assistants (my term Virtual Assistants) by Stone Temple

https://www.stonetemple.com/digital-personal-assistants-test

 
Posted : 30/05/2017 5:52 pm
jaclaz
(@jaclaz)
Posts: 5133
Illustrious Member
 

There is a study out about Personal Assistants (my term Virtual Assistants) by Stone Temple

https://www.stonetemple.com/digital-personal-assistants-test

I wonder how that "study" is related - even very indirectly - with forensics. ?

jaclaz

 
Posted : 30/05/2017 7:11 pm
Share: