python requests run js file from get

Good news: there is now a requests module that supports javascript: https://pypi.org/project/requests-html/

from requests_html
import HTMLSession

session = HTMLSession()

r = session.get('http://www.yourjspage.com')

r.html.render() # this call executes the js in the page

As a bonus this wraps BeautifulSoup, I think, so you can do things like

r.html.find('#myElementID').text

Now, all you need to do is the following code:

from selenium.webdriver.chrome.options
import Options
from selenium
import webdriver

chrome_options = Options()
chrome_options.add_argument("--headless")

driver = webdriver.Chrome(chrome_options = chrome_options)

If you do not know how to use Selenium, here is a quick overview:

driver.get("https://www.google.com") #Browser goes to google.com

Finding elements: Use either the ELEMENTS or ELEMENT method. Examples:

driver.find_element_by_css_selector("div.logo-subtext") #Find your country in Google.(singular)

its a wrapper around pyppeteer or smth? :( i thought its something different

    @property
    async def browser(self):
       if not hasattr(self, "_browser"):
       self._browser = await pyppeteer.launch(ignoreHTTPSErrors = not(self.verify), headless = True, args = self.__browser_args)

    return self._browser

the cookie generated after checking for javascript for this example is "cf_clearance". so simply create a session. update cookie and headers as such:

s = requests.Session()
s.cookies["cf_clearance"] = "cb4c883efc59d0e990caf7508902591f4569e7bf-1617321078-0-150"
s.headers.update({
   "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
   AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 89.0 .4389 .90 Safari / 537.36 "
})
s.get(url)

Suggestion : 2

To log in to this website (https://www.reliant.com) using python requests etc. (I know this could be done with selenium or PhantomJS or something, but would prefer not to),During the log in process there a couple of redirects where “session ID” type params are passed. Most of these i can get but there’s one called dtPC that appears to come from a cookie that you get when first visiting the page. As far as I can tell, the cookie originates from this JS file (https://www.reliant.com/ruxitagentjs_ICA2QSVfhjqrux_10175190917092722.js). This url is the next GET request the browser performs after the initial GET of the main url. All the methods i’ve tried so far have failed to get me that cookie.

from requests_html
import HTMLSession

url = r 'https://www.reliant.com'
url2 = r 'https://www.reliant.com/ruxitagentjs_ICA2QSVfhjqrux_10175190917092722.js'
headers = {
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
   'Accept-Encoding': 'gzip, deflate, br',
   'Accept-Language': 'en-US,en;q=0.9',
   'Cache-Control': 'max-age=0',
   'Connection': 'keep-alive',
   'Host': 'www.reliant.com',
   'Sec-Fetch-Mode': 'navigate',
   'Sec-Fetch-Site': 'none',
   'Sec-Fetch-User': '?1',
   'Upgrade-Insecure-Requests': '1',
   'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.3'
}

headers2 = {
   'Referer': 'https://www.reliant.com',
   'Sec-Fetch-Mode': 'no-cors',
   'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
}

s = HTMLSession()
r = s.get(url, headers = headers)
js = s.get(url2, headers = headers2).text

r.html.render() #works but doesn 't get the cookie
r.html.render(script = js) #fails on Network error

s = HTMLSession()
r = s.get(url, headers = headers)
print(r.status_code)

c = r.html.render(script = 'document.cookie')

c = urllib.parse.unquote(c)
c = [x.split('=') for x in c.split(';')]
c = {
   x[0]: x[1]
   for x in c
}
print(c)

Suggestion : 3

To install it, we run

pip install requests - html

Then, we write

from requests_html
import HTMLSession

session = HTMLSession()
r = session.get('http://www.example.com')
r.html.render()

Suggestion : 4

Last Updated : 17 May, 2022

Syntax:

pip install js2py

Suggestion : 5

In this tutorial, we will walk you through code that will extract JavaScript and CSS files from web pages in Python. , In this tutorial, you learned how to extract JavaScript and CSS files from web pages in Python. To extract the CSS and JavaScript files, we have used web scrapping using Python requests and beautifulsoup4 libraries. , Next, let's write a similar Python program that will extract JavaScript from the webpage. , The webpage can have multiple CSS and JavaScript files, and the more files an HTML page has, the more time the browser will take to load the complete webpage. Before we can extract JavaScript and CSS files from web pages in Python, we need to install the required libraries.

To install requests for your Python environment, run the following pip install command on your terminal or command prompt:

pip install requests

2._

pip install beautifulsoup4

In an HTML file, the CSS can be embedded in two ways, internal CSS and external CSS . Let's write a Python program that will extract the internal as well as the external CSS from an HTML file. Let's start with importing the modules:

import requests
from bs4
import BeautifulSoup

After defining the function, let's send a Get request to the webpage URL and call the page_Css() function.

#url of the web page
url = "https://www.techgeekbuzz.com/"

#send get request to the url
response = requests.get(url)

#parse the response HTML page
page_html = BeautifulSoup(response.text, 'html.parser')

#Extract CSS from the HTML page
page_Css(page_html)

A Python Program to Extract Internal and External CSS from a Webpage

import requests
from bs4
import BeautifulSoup

def page_Css(page_html):
   #find all the external CSS style
external_css = page_html.find_all('link', rel = "stylesheet")

#find all the intenal CSS style
internal_css = page_html.find_all('style')

#print the number of Internal and External CSS
print(f "{response.url} page has {len(external_css)} External CSS tags")
print(f "{response.url} page has {len(internal_css)} Internal CSS tags")

#write the Internal style CSS in internal_css.css file
with open("internal_css.css", "w") as file:
   for index, css_code in enumerate(internal_css):
   file.write(f "\n  //{index+1} Style\n")
file.write(css_code.string)

#write the External style CSS links in external_css.txt file
with open("external_css.txt", "w") as file:
   for index, css_tag in enumerate(external_css):
   file.write(f "{css_tag.get('href')} \n")
print(index + 1, "--------->", css_tag.get("href"))

#url of the web page
url = "https://www.techgeekbuzz.com/"

#send get request to the url
response = requests.get(url)

#parse the response HTML page
page_html = BeautifulSoup(response.text, 'html.parser')

#Extract CSS from the HTML page
page_Css(page_html)