Results 1 to 3 of 3

Thread: Web News Paywall Killer

  1. #1

    Default Web News Paywall Killer

    Requires python 2.7

    # Web News Paywall Killer
    # presence 2017
    # Python 2.7
    from HTMLParser import HTMLParser
    import urllib
    import re
    url = ""    
    class MLStripper(HTMLParser):
        def __init__(self):
            self.fed = []
        def handle_data(self, d):
        def get_data(self):
            return ''.join(self.fed)
    def strip_tags(html):
        s = MLStripper()
        return s.get_data()
    def strip_brackets(string):
        ret = ''
        skip1c = 0
        skip2c = 0
        skip3c = 0
        for i in string:
            if i == '{':
                skip1c += 1
            elif i == '(':
                skip2c += 1
            elif i == '[':
                skip3c += 1
            elif i == '}' and skip1c > 0:
                skip1c -= 1
            elif i == ')'and skip2c > 0:
                skip2c -= 1
            elif i == ']' and skip3c > 0:
                skip3c -= 1
            elif skip1c == 0 and skip2c == 0 and skip3c == 0:
                ret += i
        return ret
    website = urllib.urlopen(url)
    raw_data =    
    # remove html tags    
    data = strip_tags(raw_data)
    # remove anything in brackets
    data = strip_brackets(data)
    # remove if 1st character   
    for i in [';']:
        data = " ".join(filter(lambda x:x[-1]!=i, data.split()))   
    # remove if 1st character   
    for i in ['.','@',':','%',',',';','=','|','~','-','/','&','>','<','#','!','^']:
        data = " ".join(filter(lambda x:x[0]!=i, data.split()))    
    # remove if 1st two characters
    for i in ['a:','i$','a.','p:']: 
        data = " ".join(filter(lambda x:x[:2]!=i, data.split())) 
    # remove if 1st three characters    
    for i in ['div','trb','img','goo','a,.','url']: 
        data = " ".join(filter(lambda x:x[:3]!=i, data.split()))       
    # remove html
    pattern = re.compile('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+')
    data = pattern.sub('', data)
    # remove long words
    data = ' '.join([w for w in data.split() if len(w)<20])
    # remove duplicate consecutives
    data = re.sub(r'\b(\w+)( \1\b)+', r'\1', data)
    # remove duplicate consecutives
    data = re.sub(r'\b(\w+)( \1\b)+', r'\1', data)
    # remove these phrases
    for i in ['screen and']:
        data = data.replace(i, '')
    # remove white space
    data = " ".join(data.split())   
    print data

    Click image for larger version. 

Name:	Screenshot from 2017-02-16 17:15:16.jpg 
Views:	0 
Size:	58.1 KB 
ID:	5629

    >>> python

    >>> Netanyahu says Israel will never give up the Golan Heights LA Times if iframe.trb_embed p@media print, print World MIDDLE EAST Netanyahu says Israel will never give up the Golan Heights Benjamin Netanyahu Sebastian Scheiner Associated Press Israeli Prime Minister Benjamin Netanyahu, center,held a Cabinet meeting for the first time in the contestedGolan HeightsonApril 17, 2016. Israeli Prime Minister Benjamin Netanyahu, center,held a Cabinet meeting for the first time in the contestedGolan HeightsonApril 17, 2016. Joshua Mitnick Israel will never relinquish control over the Golan Heights, Prime Minister Benjamin Netanyahu vowed on Sunday during the first Cabinet meeting to be held in the contested territory. He said that the chaos from the civil war in neighboring Syria made returning the strategic plateau to Damascus too risky for Israel.“It's time that the international community recognize two fundamental facts,'' Netanyahu declared after posing with Cabinet members at a spot overlooking the Sea of Galilee Israel's largest water reservoir. “First, no matter what happens on the other side of the border, the line itself isn't going to change. Second, it’s time that the international community finally recognize that the Golan will remain forever under Israel's sovereignty.” Fourteen years after conquering the Golan in 1967, Israel's government essentially annexed the territory by extending Israeli law to the territory and giving the Syrian Druze villagers who lived there the status of permanent residents of the Jewish state.Most of the international community rejected the move and considers the Golan occupied territory.The Golan is a largely rural region, and attracts more tourists than newcomers, despite Netanyahu’s pledge to boost development there. About 20,000 Syrian Druze live in a handful of villages near the eastern cease-fire lines that are patrolled by United Nations peacekeepers, and about 25,000 Israelis live on settlements there. By contrast, hundreds of thousands of Israelis live on settlements in the West Bank. See the most-read stories this hour the story But the Golan also holds the promise of having oil, a theory that has pushed Afek Oil and Gas, a subsidiary of U.S.-based Genie Energy, to begin exploring in the area. The company has drilled for oil at four sites in the Golan since Feb. 2015, according to the Jerusalem Post.Syrian state media aired a statement from “the sons of the invaded Arabic Syrian Golan,” rebuffing Netanyahu’s declaration and saying that the disputed territory would always truly be part of Syria, calling the Israelis “invaders” and the opposition fighters would use all means to take back the Golan Heights, including military ones, Syrian Deputy Foreign Minister Faisal Mekdad said in an interview with Al Mayadeen, a Lebanese broadcaster with close ties to the Syrian government. “We have never given up on the choice of resistance and Israel wants to provoke us and we will not submit.”Since the 1990s, a succession of Israeli prime ministers had discussed returning nearly all of the Golan Heights to Syria in return for a full normalization of ties.Netanyahu’s remarks seemed to close the door to that possibility. With fighting raging in Syria, politicians in Netanyahu’s coalition have frequently argued that such a peace deal would have brought chaos to Israel’s doorstep.On Sunday, Netanyahu predicted that Syria’s warring religious groups would remain at odds for the foreseeable future and argued that Israel’s continued control over the Golan Heights would be a “stabilizing factor” in the region. While Israel has built a new fence in the Golan to prevent the conflict from spilling over its borders, it has also been providing humanitarian aid to antigovernment villages and fighters in the border region as a way of trying to bolster stability amid the power vacuum in Syria.“I doubt it if Syria will ever return to what it once was,” Netanyahu said, echoing what he said over the weekend to U.S. Secretary of State John F. Kerry. “We won’t oppose a political settlement in Syria on condition it doesn’t come at the expense of Israel. In other words, at the end of the day, the forces of Iran, Hezbollah, and ISIS must be expelled from Syrian territory.’’A militia in a village by the Israeli-Syrian border declared its allegiance to Islamic State this year, alarming Israeli intelligence officials. That militia’s offensive now threatens to destroy the Southern Front, a loose coalition of Syrian rebel factions that have received backing from the West.Since the outbreak of the Syrian civil war five years ago, Israel has avoided taking sides in the conflict, and has refrained from commenting on the potential outcome. When Israel has intervened, it's been to strike weapons shipments to Hezbollah in Iran and to keep pro-Iranian militias away from the Golan Heights line between the two countries.The Syrian government, however, insists that Israel is offering support to the rebels, even tolerating the presence of the Al Qaeda-affiliated Nusra Front as one of the dominant forces in the Golan and the southwestern province of Qunaiterah.A cache of leaked files that emerged last year also cast doubt on the idea that Israel is only providing humanitarian assistance to Syrian rebels. The personal files of Mendi Safadi, an Israeli Druze politician from the north of the country, were leaked to the Lebanese newspaper Al-Safeer by a group of hackers.The files included long-running correspondence with Syrian rebels about weapons deals and intelligence sharing, and also suggested that Safadi had tried to influence the selection of rebel political leaders based in Turkey’s southern city of Gaziantep.Safadi insisted all evidence of giving military support to the rebels was declaration risks embroiling Israel in the Syrian conflict, Itamar Rabinovich, a former Israeli peace negotiator with Syria and former ambassador to the U.S. told Israel’s public radio on Sunday. “We are putting ourselves in the center of the debate,’’ he said.Others also questioned the timing of the speech, which comes as the Syrian peace talks in Geneva sputtered on with no solution to the conflict in sight.“Geneva presents another opportunity for Netanyahu to lay down a marker and get the world accustomed to the idea that the Golan is Israeli and that it is futile for people to think of it in any other context,” said Dr. Joshua Landis, director of the Center for Middle East Studies at the University of Oklahoma and a Syria expert. “I am sure it is for local politics too.”While Netanyahu’s comments on the Golan cater to Israeli public opinion, they also signal to the international community that it must take Israel’s position into account in negotiating a solution to the Syrian civil war, said Eyal Zisser, a political science professor at Tel Aviv University who focuses on Syria. “Half a year ago we were sure the war is going to continue for a long time, and there’s no solution,’’ he said. “Now there’s a sense that there might be a settlement imposed by Russia and the Americans, and it’s important for Netanyahu to remind everyone that we are there, and they should be taking our interests into is a special correspondent. Special correspondent Nabih Bulos contributed from Beirut. ALSOSyria holds elections despite critics' contention that they undermine peace talks'Those islands are Egyptian!': Ceding of territory to Saudi Arabia prompts outragePresident Obama visits CIA headquarters as U.S. steps up attacks against Islamic State Privacy Policy Copyright 2017, Los Angeles Times Middle East Syria Israel Benjamin Netanyahu Islamic State Ex-Palm Springs mayor and 2 developers charged with corruption involving $375,000 in bribes Slaying victim Susan Berman said Robert Durst admitted to killing wife, witness testifies After Puzder debacle, Trump opts for a more conventional Labor nominee in Alexander Acosta Most An Ad 65 body, body a a, p

    so far:

    - LA Times

    - WSJ
    - Financial Times
    - Chicago Tribune
    Last edited by presence; 02-16-2017 at 06:43 PM.

    'We endorse the idea of voluntarism; self-responsibility: Family, friends, and churches to solve problems, rather than saying that some monolithic government is going to make you take care of yourself and be a better person. It's a preposterous notion: It never worked, it never will. The government can't make you a better person; it can't make you follow good habits.' - Ron Paul 1988

    Awareness is the Root of Liberation Revolution is Action upon Revelation

    'Resistance and Disobedience in Economic Activity is the Most Moral Human Action Possible' - SEK3

    Flectere si nequeo superos, Acheronta movebo.

    ...the familiar ritual of institutional self-absolution...
    ...for protecting them, by mock trial, from punishment...

  2. Remove this section of ads by registering.
  3. #2


    Amazing. Not enough reps for you. It looks like fun to tinker with too, if I find any lines that improve it I'll report back.

  4. #3


    Kewl, outta rep too...

Similar Threads

  1. Is Obama a killer?
    By enhanced_deficit in forum U.S. Political News
    Replies: 47
    Last Post: 02-08-2017, 11:31 PM
  2. Replies: 2
    Last Post: 11-15-2013, 08:41 AM
  3. Serial Killer
    By green73 in forum U.S. Political News
    Replies: 6
    Last Post: 04-28-2013, 03:52 PM
  4. FDA: A Killer Agency
    By FrankRep in forum Health Freedom
    Replies: 4
    Last Post: 07-14-2011, 04:51 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts