Selenium with PhantomJS: Yahoo login form not submitting (Python bindings)

Question:

Im writing a python 2.7 script using selenium webdriver on OS X to login to Yahoo fantasy sports and automate some actions.

The script works fine with webDriver Firefox and Chromedriver. I’ve recently started using the PhantomJS (GhostDriver) and I’ve found I can’t get the PhantomJS Selenium Driver (GhostDriver) to log into Yahoo login forms.

#!/usr/bin/python
import time
from selenium import webdriver
from selenium.webdriver import PhantomJS
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from sys import argv
import click

@click.command()
@click.option('--days', type=int, prompt='Number of days to set active lineup', help='Number of days to set active lineup')
@click.option('--username', prompt='Your Yahoo username:', help='Your Yahoo account username')
@click.option('--password', prompt='Your Yahoo passwordname:', help='Your Yahoo account password')
def start_active_players(days, username, password):
    """Simple python program that sets your active players for the next number DAYS."""
    print("Logging in as: " + username)

    dcap = DesiredCapabilities.PHANTOMJS.copy()
    dcap['javascriptEnabled'] = True 
    dcap['browserConnectionEnabled'] = True 
    dcap['acceptSslCerts'] = True
    dcap['localToRemoteUrlAccessEnabled'] = True 
    dcap['webSecurityEnabled'] = True 
    dcap['version'] = ''


    driver = webdriver.PhantomJS(executable_path='/Users/devin.mancuso/node_modules/phantomjs/bin/phantomjs', desired_capabilities=dcap)

    driver.get('https://login.yahoo.com/config/login?.src=spt&.intl=us&.done=http%3A%2F%2Fbasketball.fantasysports.yahoo.com%2Fnba')

    with open('jquery-2.1.3.min.js', 'r') as jquery_js: jquery = jquery_js.read() #read the jquery from a file
    driver.execute_script(jquery) #active the jquery lib

    driver.find_element_by_id('login-username').send_keys(username) 
    driver.find_element_by_id('login-passwd').send_keys(password)
    driver.implicitly_wait(8) # 8 seconds
    driver.find_element_by_name('signin').click()
    #form1 = driver.find_element_by_id('mbr-login-form')
    #form1.submit()
    driver.implicitly_wait(8) # 8 seconds
    driver.save_screenshot('screenshot.png')

    driver.find_element_by_xpath("//a[text() = 'My Team ']").click()
    driver.implicitly_wait(8) # 8 seconds

    for x in range(0, days):

        driver.find_element_by_xpath("//a[text() = 'Start Active Players']").click()
        driver.implicitly_wait(2) # 2 seconds
        date_text = driver.find_element_by_xpath("//span[@class='flyout-title']").text
        print("Starting active players for: " + date_text)
        driver.find_element_by_xpath("//a[contains(@class, 'Js-next')]").click()
        driver.implicitly_wait(2) # 2 seconds

    driver.quit()

if __name__ == '__main__':
    start_active_players()

The script fails on line 47,

driver.find_element_by_xpath(“//a[text() = ‘My Team ‘]”).click()

when it attemps to find the link with the text My Team. A screenshot dump shows that it never makes it past the login form. An on-screen error message above the form states

Please reload the page and try again or use another browser

I saw in this post and thus included the execute_script command to load in Jquery locally, but that didn’t solve it. I’m not sure if it’s a Yahoo security issue that is stopping PhantomJS. But why would it only fail on a headless browser and not FF or Chrome?

I also found this question and have tried to submit the form itself instead of clicking the button, but that made no difference. I’ve commented out the code in the example above.

PhantomJS version: 2.0.0

Asked By: Devin

||

Answers:

Solution was to set the PhantomJS userAgent using the python bindings. Discovered through Andrew Magee’s recommendations in comments and via this conversation on the ghostdriver github.

DesiredCapabilities.PHANTOMJS['phantomjs.page.settings.userAgent'] = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:16.0) Gecko/20121026 Firefox/16.0'

driver = webdriver.PhantomJS(executable_path='/Users/devin.mancuso/node_modules/phantomjs/bin/phantomjs')
Answered By: Devin