Can Selenium be used to retrieve a list of pinned IDs from a website?

I am currently developing a web scraper that is required to open multiple tabs of items with filled icons. Specifically, each page that needs to be opened contains the div class="course-selector-item-pinned" in its source code.

<dropdown-content max-width="800" min-width="450" no-padding="" vertical-offset="0" dir="ltr" dropdown-content="" style="--dropdown-verticaloffset:0px;" opened=""><div class="classselector-wrapper" aria-live="assertive">
    <div id="classSelectorId" class="placeholder placeholder-live" aria-live="assertive">
        <div class="2_7_615 2_8_459 body-compact">
            <ul class="datalist vui-list">
                <li class="datalist-item datalist-item-actionable datalist-simpleitem vui-selected" id="2_9_421" data-actionid="2_11_656">
                    <div class="datalist-item-content" title="Class 1">
                        <div class="class-selector-item class-selector-item-pinned" data-org-unit-id="12345">
                            <div class="2_160_610 class-selector-item-name">
                                <a class="link datalist-item-actioncontrol" id="2_11_656" href="/abc/home/12345">Class1</a>
                            </div>
                            <span id="2_10_630" data-active-id="2_161_292" data-inactive-id="2_162_883"><button-icon icon="tier1:pin-filled" id="2_161_292" onclick="O(&quot;__g2&quot;,3)();" text="Un-pin &quot;Class 1&quot;" dir="ltr" type="button"></button-icon>
                            <button-icon icon="tier1:pin-hollow" class="hidden" id="2_162_883" onclick="O(&quot;__g2&quot;,4)();" text="Pin &quot;Class 1&quot;" dir="ltr" type="button"></button-icon>
                            </span></div>
                    </div>
                    <div class="clear"></div>
                </li>
                <li class="datalist-item datalist-item-actionable datalist-simpleitem vui-selected" id="2_12_929" data-actionid="2_14_114">
                    <div class="datalist-item-content" title="Class 2">
                        <div class="class-selector-item class-selector-item-pinned" data-org-unit-id="23456">
                            <div class="2_160_610 class-selector-item-name">
                                <a class="link datalist-item-actioncontrol" id="2_14_114" href="/abc/home/23456">Class 2</a>
                            </div>
                            <span id="2_13_229" data-active-id="2_163_477" data-inactive-id="2_164_80"><button-icon icon="tier1:pin-filled" id="2_163_477" onclick="O(&quot;__g2&quot;,5)();" text="Un-pin &quot;Class 2&quot;" dir="ltr" type="button"></button-icon>
                            <button-icon icon="tier1:pin-hollow" class="hidden" id="2_164_80" onclick="O(&quot;__g2&quot;,6)();" text="Pin &quot;Class 2&quot;" dir="ltr" type="button"></button-icon>
                            </span></div>
                    </div>
                    <div class="clear"></div>
                </li>
                <li class="datalist-item datalist-item-actionable datalist-simpleitem vui-selected" id="2_15_372" data-actionid="2_17_26">
                    <div class="datalist-item-content" title="Class 3">
                        <div class="class-selector-item class-selector-item-pinned" data-org-unit-id="34567">
                            <div class="2_160_610 class-selector-item-name">
                                <a class="link datalist-item-actioncontrol" id="2_17_26" href="/abc/home/34567">Class 3</a>
                            </div>
                            <span id="2_16_595" data-active-id="2_165_349" data-inactive-id="2_166_873"><button-icon icon="tier1:pin-filled" id="2_165_349" onclick="O(&quot;__g2&quot;,7)();" text="Un-pin &quot;Class 3&quot;" dir="ltr" type="button"></button-icon>
                            <button-icon icon="tier1:pin-hollow" class="hidden" id="2_166_873" onclick="O(&quot;__g2&quot;,8)();" text="Pin &quot;Class 3&quot;" dir="ltr" type="button"></button-icon>
                            </span></div>
                    </div>
                    <div class="clear"></div>
                </li>

The task for the web scraper involves identifying all div classes with "course-selector-item-pinned" and extracting the values from data-org-unit-ids. For example, the expected result would be [12345, 23456, 34567] in this scenario.

The specific line of source code referred to is:

<div class="class-selector-item-pinned" data-org-unit-id"12345">
<div class="class-selector-item-pinned" data-org-unit-id"23456">
<div class="class-selector-item-pinned" data-org-unit-id"34567">

Despite initial attempts, the current implementation does not return any results.

Get List of Unit IDs

courseString='https://example.com/abc/p/home'
listofUnitID =[]
links = [elem.get_attribute("data-org-unit-id") for elem in driver.find_elements_by_class_name("class-selector-item-pinned")]

Filter out none type from list

res = []
for val in links: 
    if val != None : 
        res.append(val) 
print(res)

List to Keep Only Classes

for i in res:
    if courseString in i:
        listOfHref.append(i)
        print(listOfUnitID)
htmljquerycssseleniumselenium-chromedriver

Answer №1

If you examine the HTML code, it appears that you are attempting to scrape the specific div below:

<div class="class-selector-item class-selector-item-pinned" data-org-unit-id="34567">
...
...

However, make sure not to target these elements:

<div class="class-selector-item-pinned" data-org-unit-id="12345">
<div class="class-selector-item-pinned" data-org-unit-id="23456">
<div class="class-selector-item-pinned" data-org-unit-id="34567">

This mistake will result in no data being returned because you are selecting a div with multiple classes.

Remember that .find_elements_by_class_name is meant for single class names only.

You should consider using 

.find_elements_by_css_selector('css_selector')
, like so:

links = [elem.get_attribute("data-org-unit-id") for elem in driver.find_elements_by_css_selector(".class-selector-item.class-selector-item-pinned")]

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

confirmation message upon completing a form submission

<script> var remainingCredit = document.getElementById("cor_credit"); var remaining = document.getElementById("remain_credit"); function validateForm() { if (remaining.value < remainingCredit.value) { return conf ...

javascripthtml

Increase the spacing between rows and columns in Bootstrap

Is there a way to create gaps between rows and columns in bootstrap without disrupting the layout? I am trying to have three columns on each row by using col-4, but when I try to add margin or padding, it messes up the design. My goal is to achieve a layo ...

cssbootstrap-5

Jenkins is having trouble running Selenium tests with the FirefoxDriver, even though they work perfectly on a local desktop setup

I am currently facing an issue while trying to execute my Selenium tests in Jenkins. These tests are running smoothly on my local machine using Maven test or IntelliJ. Here is the setup I am using: Firefox 39 x64 Selenium 2.46.0 SeleniumHQ pl ...

seleniumjenkinsselenium-firefoxdriver

Unable to transform Table layout into DIVs

My form for data input consists of labels with inputs, but the labels are localized so I am not sure about the length of text for different languages. To tackle this issue, I initially used a table layout: <table> <tr> <td> ...

htmlcsshtml-table

Incorrect .offset().top calculation detected

Within a webpage, I have multiple sections. Positioned between the first and second section is a navbar menu. As the user scrolls down and the navbar reaches the top of the page, a function is triggered to fix it in place at the top. While this functionali ...

javascriptjqueryhtmlcss

Place the dataLabel above a specified column in Highcharts

My bar chart has a stacked design, with an invisible bar added to the height of the tallest bar to ensure clickability even for small values. Without this invisible stack, columns with a value of 1 would be difficult to click on. One issue I am facing is ...

jqueryhighcharts

Arranging an image to appear directly underneath another image upon hovering the mouse over it

I want image two to appear below image one when the mouse hovers over image one. <ul> <li> <img id="img1" src="imageone.png"> <br> <img id="img2" src="imagetwo.png"> </li> </ul> Any assistance ...

htmlcss

Error with AngularJS: IE not showing dropdown options for MultiSelect

My application features a multiselect dropdown menu that shows a list of countries. While this dropdown functions correctly in Chrome, it displays options differently in IE as shown below: https://i.stack.imgur.com/xhOmV.png I have attempted to adjust th ...

javascripthtmlangularjs

Tips on incorporating background color into HTML emails dispatched via PHP mail()

Struggling to add a background color to my HTML email. I've attempted various methods: Inserting <script> tags with CSS code inside Adding bgcolor in body tags Using a CSS style sheet All to no avail. I suspect it could be the email provi ...

phphtmlcsshtml-emailemail

Tips for customizing the jQuery countdown styling

Currently, I have incorporated the jQuery countdown into my project which consists of solely the essential Javascript required for the countdown to function. However, I am interested in achieving a similar visually appealing "animated" style similar to t ...

jquerycssjquery-countdown

How can I determine the mouse's location relative to the bottom right corner of the

Currently, I am in the process of writing jQuery code that will detect if the mouse is positioned near the bottom or right side of a window, and then adjust the tooltip's position accordingly. This becomes crucial because the table in which these tool ...

jqueryhtmlcss

Vuetify Container with a set maximum width that remains fixed

I recently started developing a web application using Vue.js and Vuetify (https://vuetifyjs.com/en/). Currently, I have a basic layout with 3 columns. However, I noticed that the width of these columns is restricted to a maximum of 960px due to some defau ...

cssvue.jsvuetify.js

Fuel Calculation - Unable to pinpoint the error

My JavaScript Code for Calculating CO2 Emissions I'm working on a program that calculates how much CO2 a person produces per kilometer. This is part of my school project and I'm still learning, so please bear with me... I also just signed up on ...

javascripthtmlbootstrap-4

Selenium continuously clicks the button in a loop

Need help scraping bike details from this page: The page only displays bike names without prices. I'd like to click on each bike, then click "Shop Now" to navigate to individual pages and get the current price (as the bikes switch periodically). How ...

pythonseleniumweb-scrapingweb-crawler

Can a border-bottom be made the same size as a class it is not linked to in CSS?

Can I set a border-bottom to match the size of a separate class? For example, I want a border-bottom at the bottom of each <section> tag. However, since the sections span the full width of the screen, the border-bottom inherits that width. Each < ...

htmlcsssassbootstrap-4

Combine various pieces of information into a single form using JSON formatting

[ { "name": "Test", "type": "Private", "item":[{"itmeNo":"PT-15003C","quantity":"3"}, {"itmeNo":"PT-15003C","quantity":"3"}], "successMsg":"Item(s) added to the job list." } ] Hello there, I am currently working on data parameter ...

javajsonseleniumparameterization

Navigating through various frames with Selenium in conjunction with Java

Is it possible to send keys simultaneously to Card Number, Expiration Date, and CVV text fields within an iframe? In my testing experience, I have noticed that whichever frame I switch to first in the test case gets located and receives the keys, while th ...

javaseleniumselenium-webdriveriframeframe

After minimizing the window, the jQuery slider animation sped up significantly

I am currently using a coda slider on my website, but I have encountered an issue where the content slider plays too fast after the window is minimized. I believe this may be related to the use of SetTimeout, but I have not been able to find a perfect so ...

javascriptjqueryanimationtimeoutslider

If the URL hash matches the ID of a specific div, then take action on the div's child element

Imagine I have the following HTML structure: <div class=".blog-item-holder"> <div id="AAAA"><div class="inner">AAAA</div></div> <div id="BBBB"><div class="inner">BBBB</div></div> <div id="CCCC">& ...

jqueryhtmlcssurlfragment-identifier

Deciphering HTML with AngularJS

We are currently working with AngularJS version 1.5.6 and facing an issue with HTML characters not being displayed correctly in our text. Despite trying various solutions, we have been unsuccessful in resolving this issue. We have explored numerous discuss ...

javascripthtmlangularjsdecode