Like many of you, over the last 18 months we have seen a huge shift in how our staff are working, people are at home, people working remotely permanently, or being unable to get into their regular office. That has meant a shift in your detections, previously you had people lighting up internal firewalls, or you saw events on your internal Active Directory, now you are also interested in cloud service access, suspicious MFA events or VPN activity.
With this change we unsurprisingly noticed a dramatic uptick in identity related alerts – users connecting from new counties, or via anonymous IP addresses, unfamiliar properties and impossible travel events. When we get these kind of events (even for failed attempts), we proactively log our users our of Azure AD to make them re-authenticate + MFA, it isn’t perfect but it’s an easy automation that doesn’t annoy anyone too much and buys some time for a cyber security team member to check out the detail. For each of these events we also populate the Azure Sentinel incident with the last 10 sign-ins for the user affected (excluding any from a trusted location) for someone to investigate.
SigninLogs
| where UserPrincipalName == "attackeduser@yourdomain.com"
| where IPAddress !startswith "10.10.10"
| project TimeGenerated, AppDisplayName, ResultType, ResultDescription, IPAddress, Location, UserAgent
| order by TimeGenerated desc Around January of this year we noticed a number of users showing really strange behaviour, they would have one or two wrong password attempts (ResultType 50126) flagged on their account from a location the user had never logged in from, and no other risk detections, just one or two attempts and that was it. After we noticed a half dozen the same, we decided to look a bit closer and noticed the same user agent being used for all the attempts, so we dug into the data. We looked for all sign in data from that agent, and bought back the user, the result, what application was being accessed, the IP and the location.
SigninLogs
| where UserAgent contains "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"
| project UserPrincipalName, ResultType, AppDisplayName, IPAddress, Location
The data looked a bit like this – lots of attempts on different users, very rarely the same IP address or location twice in a row, locations not expected for our business, maybe two attempts at a user at most and then move on, and thankfully none successful, only wrong passwords (50126) and account locks (50053). We also noticed a second UserAgent with the same behaviour so we added that to the query and found more hits in much the same pattern.
SigninLogs
| where UserAgent contains "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148" or UserAgent contains "Outlook-iOS/723.4027091.prod.iphone (4.28.0)"
| project TimeGenerated, UserPrincipalName, ResultType, AppDisplayName, IPAddress, LocationOver a 6 month period, it was pretty low noise, peaking at 34 attempts in a single day, but usually less than 10.
We also double checked and there were no legitimate sign in activities from these UserAgents, only suspect ones. Some users were being targeted by one UserAgent, some the other and some by both, to detect those being targeted by both, you can use a simple join in KQL.
let agent1=
SigninLogs
| where UserAgent contains "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"
| distinct UserPrincipalName;
let agent2=
SigninLogs
| where UserAgent contains "Outlook-iOS/723.4027091.prod.iphone (4.28.0)"
| distinct UserPrincipalName;
agent1
| join kind=inner agent2 on UserPrincipalName
| distinct UserPrincipalNameFrom January until now we have had around 1500 attempts from these UserAgents, targeting around 300 staff, with about 50 being targeted by both suspicious UserAgents.
Now that data is interesting for us cyber security people, but at the end of the day any Azure AD tenant is going to get some people knocking on the door and there isn’t much you can do about it, these IP addresses change so frequently that blocking them isn’t especially practical. Of the 1500 attempts we have seen about 660 different IP addresses. What we did do is configure an Azure Sentinel analytics rule to tell us if we got a successful sign in from one of these agents. The rule is straight forward, look for the UserAgent and any successful attempts.
let successCodes = dynamic([0, 50055, 50057, 50155, 50105, 50133, 50005, 50076, 50079, 50173, 50158, 50072, 50074, 53003, 53000, 53001, 50129]);
SigninLogs
| where UserAgent contains "Outlook-iOS/723.4027091.prod.iphone (4.28.0)" or UserAgent contains "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"
| where ResultType in (successCodes)
| project UserPrincipalNameImportantly when we talk about success in Azure AD, we aren’t just interested in ResultType = 0. When we think about the flow of an Azure AD sign in, we can successfully sign in and then be blocked or stopped elsewhere. For instance 53003 means the sign on was stopped by conditional access. However, conditional access policies are applied after credentials have been validated, so in the case of an attacker, if they are blocked by conditional access it still means they have your users correct credentials. 50158 is another good example, which means an external security challenge failed (such as Ping Identity, Duo or Okta MFA), but the same logic applies – username and password are validated, then the user is directed to the third party security challenge. So for an attacker to get to that point, again they have the correct username and password. The KQL above has a list of everything that could be deemed a ‘success’.
We left this query running for around 8 months with no action, occasionally checking ourselves if the users were still be targeted, and they were. Finally last week we got a hit, a user had been phished (no one is perfect!) and the attackers signed into the account, thankfully they were stopped by a conditional access policy blocking sign ins from the particular country they tried on that attempt. We contacted the user, reset their credentials, sent them some phishing training and away they went.
In the scheme of Azure AD globally, 1500 attempts to a single tenant over the course of 8 months is not even a rounding error, Microsoft is evaluating millions of sign ins an hour and this traffic isn’t likely to flag anything special at their end. It was suspicious to our business though, and that is where your knowledge of your environment combined with the tools on offer is where you add real value.
If you are interested more generally in how often you are seeing new UserAgents for your users you can use the below query, we create a set of known UserAgent for each user over a learning period (14 days), then join against the last day (and exclude known corporate IP’s)
let successCodes = dynamic([0, 50055, 50057, 50155, 50105, 50133, 50005, 50076, 50079, 50173, 50158, 50072, 50074, 53003, 53000, 53001, 50129]);
let isGUID = "[0-9a-z]{8}-[0-9a-z]{4}-[0-9a-z]{4}-[0-9a-z]{4}-[0-9a-z]{12}";
let lookbacktime = 14d;
let detectiontime = 1d;
let UserAgentHistory =
SigninLogs
| project TimeGenerated, UserPrincipalName, UserAgent, ResultType, IPAddress
| where TimeGenerated between(ago(lookbacktime)..ago(detectiontime))
| where ResultType in (successCodes)
| where not (UserPrincipalName matches regex isGUID)
| where isnotempty(UserAgent)
| summarize UserAgentHistory = count() by UserAgent, UserPrincipalName;
SigninLogs
| where TimeGenerated > ago(detectiontime)
| where ResultType in (successCodes)
| where IPAddress !startswith "10.10.10"
| where not (UserPrincipalName matches regex isGUID)
| where isnotempty(UserAgent)
| join kind=leftanti UserAgentHistory on UserAgent, UserPrincipalName
| distinct UserPrincipalName, AppDisplayName, ResultType, UserAgentUserAgents can update quite often, mobile devices getting small updates, browsers being patched, but like everything, getting to know what is normal in your environment and detecting outside of that is half the battle won.
Microsoft Sentinel 101 