Tips for changing a non-standard tag's attribute in Python 3

Below are some tags that use non-standard attributes with the style "display: none". Since these cannot be parsed, I need to either remove style="display: none;" completely or change it to style="display: inline;".

...
<section id="box3" class="nodisp_zero" style="display: none;">
    <h1 id="box_ttl3" style="display: none;"></h1>
    <img style="width: 100%; display: none;" id="box_img3" alt="box3" src="https://smtgvs.weathernews.jp/s/topics/img/dummy.png" class="lazy" data-original="https://smtgvs.weathernews.jp/s/topics/img/201808/201808220015_box_img3_A.jpg?1533975785">
    <figcaption id="box_caption3" style="display: none;"></figcaption>
    <div class="textarea clearfix">
        <h2 id="box_subttl3" style="display: none;"></h2>
        <div class="fontL" id="box_com3" style="display: none;"></div>
    </div>
</section>
...

I attempted to utilize this code, but encountered an error stating 

TypeError: 'NoneType' object is not callable
. What steps can I take to resolve this issue?

driver.get(href)
soup_level2 = BeautifulSoup(driver.page_source, 'lxml')
soup_level2 = soup_level2.replace(r'display:\s*none', "")
images = soup_level2.find_all('img')
cssregexpython-3.xselenium

Answer №1

To eliminate the style attribute, you can do the following:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_doc, 'html.parser')
for tag in soup.findAll(lambda tag:tag.has_attr('style')):
  tag["style"] = tag["style"].replace("display: none;", "")

See Demo

Alternatively, you can use a basic regex replace method:

html_doc = re.sub(r"display:\s*none;?", "", html_doc, 0)

Check out the Demo here

Various methods exist to ensure content is fully loaded before proceeding with Selenium, such as:

element_present = EC.presence_of_element_located((By.ID, 'element_id'))
    WebDriverWait(driver, timeout).until(element_present)

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Issue with Datatables Layout - Button and Dropdown Alignment

This is my current layout in the image below. I have annotated the picture to demonstrate the desired layout changes, but I am struggling to implement them. Everything below the "New Post" button (buttons, pagination, search, table, etc.) is generated by D ...

jqueryhtmlcssdatatables

Table Collapse with Bootstrap Framework

I am currently facing an issue where I can only collapse a single row when clicking on the parent row. However, my requirement is to be able to close multiple rows simultaneously. Here is the code I am using, but I am struggling to figure out how to colla ...

htmlcsstwitter-bootstrap

What could be causing the error in this PHP regex?

Can anyone assist me with Regex? I am looking to substitute [url=http://youtube.com]YouTube.com[/url] with <a href="http://youtube.com">YouTube.com</a> using the following regex pattern: preg_replace("/[url=(.*?)](.*?)[/url]/is", '&l ...

phpregex

Adjust the size of a div by clicking a button using Javascript

<script type="text/javascript"> var btn = document.getElementById("btn"); var panel = document.getElementById("panel"); var btn2 = document.getElementById("btn2"); function expandPanel() { btn.style.display="none"; panel.style="overflow:hidd ...

javascriptphphtmlcss

The line directly following the first line within the <li> element is located in the same position

I need help with customizing the bullet background-image for an <li> element. I want the text in the bullet to be displayed outside the marker using the CSS property list-item-position: outside. However, my current implementation doesn't seem to ...

htmlcssflexbox

The header grid will disappear in the p-dialog when the browser is minimized

Check out this informative article on creating a dynamic dialog box using PrimeNG here Take a look at the image below to see how the dialog box appears when opened: https://i.sstatic.net/nUq1d.png One issue I am facing is that the grid header gets hidd ...

htmlcssangulardialogprimeng

JavaScript's addition of CSS animation not functioning as intended

I am facing an issue with a web page utilizing Bootstrap 5. The webpage consists of one element stacked on top of another. My goal is to fade-out the top element after a certain period. Here is what I have tried: HTML <div id="host" class=&qu ...

css

Concealing input special characters with Angular 8

Is there a way to display credit card numbers in an input field with a bullet special character look? 4444 •••• •••• 4444 I'm attempting to achieve this using a pipe in Angular without relying on any input mask plugins. Here's w ...

regexangularspecial-charactersinput-mask

The elimination of margin-right is not taking place

On mobile devices, I am experiencing an unwanted margin-right on my page. Despite trying various solutions found in this discussion thread, including using body, html {margin:0;} and margin-right:-8px; with !important, the default margin persists. https:/ ...

jqueryhtmlcsstwitter-bootstrap-3

Incorporating Computer Modern Serif into Your Jekyll GitHub Page

I have customized a version of the Jekyll GitHub page from here and am trying to incorporate Computer Modern Serif font as the main body font across all pages. Following instructions from an informative answer, I have downloaded the necessary files from th ...

htmlcssjekyllgithub-pageswebfonts

Is it possible to modify the sub/child divs within a draggable parent div and assign them a different class?

Greetings, after being a long-time reader, I have finally decided to post for the first time. In the process of creating a webpage with jQuery drag and drop divs, I am curious about how to change the class of a child div within a draggable div once it is d ...

javascriptjqueryhtmlcssjquery-ui

Display the navigation controls in the CSS Bootstrap Carousel only when hovering over the control section, rather than when hovering over the entire carousel

My carousel displays full page images, but I want the controllers to only appear when the mouse hovers over the specific area they are in, rather than appearing when the mouse is anywhere on the page. .carousel .carousel-control { visibility: hidden; ...

csstwitter-bootstrap

Using regular expressions to validate and extract information from a text string

I am in the process of developing two regular expressions in .NET C#. To start, I need to validate that the user input follows a specific format, which can be repeated multiple times: For example: /TEXT;dd-mm-yyyy;TEXT/TEXT;dd-mm-yyyy;TEXT/TEXT;dd-mm-y ...

c#.netregex

Python - Encountering issues when attempting to establish a connection with Selenium's standalone server using htmlunitdriver

I'm attempting to utilize a selenium remote standalone server with an htmlUnit driver using a python script. To begin, I launch the standalone server with the htmlUnit driver java -cp htmlunit-driver-2.35.1-jar-with-dependencies.jar -jar selenium-s ...

javapythonselenium

Finding an element that lacks both a class and an id, and is not consistently present - what's the trick?

Currently, I am faced with a predicament in my code where a <li> element only appears under specific conditions, making it difficult to apply positioning. This element lacks an id and class attribute, which prevents me from targeting it accurately us ...

javascripthtmlcss

CSS3's auto property for margin

This is a snippet of my HTML and CSS code that I'm working on: <header id="main"> <a href="#" id="exit"></a> <a href="http://www.reddit.com/" id="reddit"></a> <a href="http://www.stackoverflow.com/" id="st ...

htmlcssmarginpadding

Tips for merging identical media queries into a single query using SASS

Previously, I organized my media queries in LESS by combining them at a single place like this LESS: .media-mixin(@break) when (@break = 1200px) { .abc{ color: red; } } .media-mixin(@break) when (@break = 1200px) { .xyz{ color: yellow; ...

htmlcsssasslessscss-mixins

designing a personalized two-row navbar layout in Bootstrap 4

I have been attempting to design a unique custom navigation bar that I haven't come across anywhere else online. Despite my diligent search for examples or solutions, I have not been able to find what I am looking for. Therefore, I am reaching out her ...

jqueryhtmlcsstwitter-bootstrapflexbox

Creating a C# asp.net Regular expression specifically designed to work with various formats such as value, *value, value* and *value

Looking to create a regular expression in .net that allows for the following 4 types of input: 1) value 2) *value 3) value* 4) *value* The 'value' consists of a mix of digits and alphabets such as aa123, 123aa, ab12ba, a12a33. It can also be so ...

c#asp.net.netregex

encountering an error when querying MongoDB due to an invalid document issue

i encountered an issue when querying my mongodb database: bson.errors.InvalidDocument: documents must have only string keys, key was function location at 0x7f50be59ac80, the structure of my database collection can be viewed in the image below https://i.ss ...

python-3.xmongodbvariablesflaskpymongo