using a python variable in xpath selector in scrapy

Did you try:

foo = "bar"
response.xpath('//*[contains(text(), "' + foo + '")]')

use double quotes and single quotes to your advantage like this:

foo = "bar"
response.xpath("//*[contains(text(), '" + foo + "')]")

What about this :

foo = "bar"
response.xpath('//*[contains(text(), \'' + foo + '\')]')

Suggestion : 2

It matches the nodes according to the xpath query and provides the results as SelectorList instance. The parameter query specifies the XPATH query to be used.,It uses the .xpath() method for the elements and provides the results as SelectorList instance. The parameter query specifies the arguments as defined in the Selector.xpath() method.,It uses the .css() method for the elements and gives back the results as SelectorList instance. The parameter query specifies the arguments as defined in the Selector.css() method.,It supplies the CSS selector and gives back the SelectorList instance. The parameter query specifies CSS selector to be used.

For instance −

from scrapy import Selector
val = Selector(text = '<a href="#">More Info<strong>click here</strong></a>')

If you are converting a node-set to a string, then use the following format −

>> val.xpath('//a//text()').extract()

It will display the element as −

[u 'More Info', u 'click here']

It results the element as −

[u 'More Info']

The following line displays all the first li elements defined under their respective parents −

>> res("//li[1]")

Suggestion : 3

Following successful execution of the code, it is recommended that we close and quit the driver to free up system resources. The close() method terminates the loaded browser window. The quit() method ends the WebDriver application.,The way of locating a web element we have adopted is unable to identify the desired element as it is not within the browser’s Viewport. ,Now that we have completed the setup steps, we can proceed. We have created this dynamic complete search form webpage to run our scraper against. We can start by loading the example page.,When the element we locate does not exist in the DOM, use try-except event handler to avoid the termination of the program:

from selenium
import webdriver

driver = webdriver.Chrome('YOUR_PATH_TO_chromedriver.exe_FILE')

form_url = "https://iqssdss2020.pythonanywhere.com/tutorial/form/search"
driver.get(form_url)

driver.close()
driver.quit()

find_element_by_id() and find_elements_by_id() methods:
Return an element or a set of elements that have matching ID attribute values. The find_elements_by_id() method returns all the elements that have the same ID attribute values. Let’s try finding the search button from the example website. Here is the HTML code for the search button with an ID attribute value defined as search. We can find this code if we Inspect the site and reach this element in its DOM.

<input type="submit" id="search" value="Search" name="q" class="button" />

2._

search_button = driver.find_element_by_id("search")

search_button = driver.find_element_by_id("search")

search_button = driver.find_element_by_name("q")