Random User Agent

4 replies [Last post]
Od0_
Od0_'s picture
Offline
SX Inductee
Joined: 2016/08/09

Hello all,

So kind of separate from another project I am working on I threw together a quick script to randomize the User Agent in the HTTP header when making web requests. I'm going to have to integrate it into my networking project at some point. That way when I'm doing things like crawling websites I'm able to mask my identity. This will most likely morph into a software proxy of some kind. Still unclear on that. Feel free to check it out and let me know what you think. I'm still working on printing out the HTTP header information, so it's not completely done.

Keep in mind there is a companion file with 1000 different user agents all imported as strings.

import random
import requests

user_agents = []
user_agents_file = open("user_agents.txt", "r") # Text file of user agents

for agents in user_agents_file:
    # cycles through the list adds them to the user_agents array

    user_agents.append(agents)
    random_agents = random.choice(user_agents)[1:-2] # Takes out parens and new line character, also randomizes array choice

url = "http://quotes.toscrape.com"
headers = {"User-Agents" : random_agents}

r = requests.get(url, headers=headers)

print r.content