This article provides solutions to an issue where Python scripts might be detected as web crawlers when making Microsoft Graph requests.
Symptoms
A Python script that makes a Microsoft Graph request might sometimes be detected by the gateway as web crawlers. If Python scripts use a pool manager, the following error message is returned when you block the request:
{'error': {'code': 'UnknownError', 'message': '\r\n403 Forbidden\r\n\r\n
403 Forbidden
\r\n
Microsoft-Azure-Application-Gateway/v2
\r\n\r\n\r\n', 'innerError': {'date': '{UTC Date/Time}', 'request-id': '{guid}', 'client-request-id': '{guid}'}}}
Cause
The issue occurs because some Python scripts might not structure their requests in a way that conforms to the expected patterns. As a result, the gateway mistakenly identifies the requests as coming from a web crawler.
Solution
To resolve this issue, use the Microsoft Graph SDK for Python. If you don't want to use it, structure your requests similarly to how the SDK handles them by using Python's Session object to send requests.
Here's an example of how you can structure your requests manually:
from requests import Request, Session
def example_request(url):
http = Session()
req = Request('GET', url, headers=h)
prepped = req.prepare()
resp = http.send(prepped)
return resp.json()
Note:
- The
User-Agentstring in the HTTP request header should be unique to your application or script, differentiating it from generic traffic.- Including additional headers such as
Accept: application/jsoncan help clarify the intent of your request.- Incorrectly identified as crawler traffic might result in throttling or other automated mitigations by Microsoft's backend systems. Properly identifying your client helps avoid these issues.
If you use a different HTTP client, ensure that similar headers are set appropriately. Refer to your HTTP client's documentation for customizing request headers.
More information
For more guidance on best practices when using Microsoft Graph, see the following articles:
Source: Microsoft Learn / IT Pro troubleshooting by Microsoft and contributors. Documentation licensed CC BY 4.0. Code samples are covered by the MIT License. Converted from Markdown, with links normalized and embedded media omitted.
