Archive | Big Data

Why we average polls

September 19, 2016 - Ames, IA.

Here’s the NYT pollster, who gave the same set of data to four different pollsters. Guess what happened?

Well, well, well. Look at that. A net five-point difference between the five measures, including our own, even though all are based on identical data. Remember: There are no sampling differences in this exercise. Everyone is coming up with a number based on the same interviews.

Their answers shouldn’t be interpreted as an indication of what they would have found if they had conducted their own survey. They all would have designed the survey at least a little differently – some almost entirely differently.

But their answers illustrate just a few of the different ways that pollsters can handle the same data – and how those choices can affect the result.

So what’s going on? The pollsters made different decisions in adjusting the sample and identifying likely voters. The result was four different electorates, and four different results.

I remember trying to explain polling variables to C&L readers, and got raked over the coals by people calling me a shill because I said polls showed Hillary winning.

If you’re not paranoid

Tin Foil Hat

You’re crazy.

Adultery, divorce. I saw a pattern here, one that I found especially unwelcome because at the time I was recently engaged. Evidently, some callous algorithm was betting against my pending marriage and offering me an early exit. Had merely typing seduction into a search engine marked me as a rascal? Or was the formula more sophisticated? Could it be that my online choices in recent weeks—the travel guide to Berlin that I’d perused, the Porsche convertible I’d priced, the old girlfriend to whom I’d sent a virtual birthday card—indicated longings and frustrations that I was too deep in denial to acknowledge? When I later read that Facebook, through clever computerized detective work, could tell when two of its users were falling in love, I wondered whether Google might have similar powers. It struck me that the search engine might know more about my unconscious than I do—a possibility that would put it in a position not only to predict my behavior, but to manipulate it. Lose your privacy, lose your free will—a chilling thought.

Around the same time, I looked into changing my car-insurance policy. I learned that Progressive offered discounts to some drivers who agreed to fit their cars with a tracking device called Snapshot. That people ever took this deal astonished me. Time alone in my car, unobserved and unmolested, was sacred to me, an act of self-communion, and spoiling it for money felt heretical. I shared this opinion with a friend. “I don’t quite see the problem,” he replied. “Is there something you do in your car that you’re not proud of? Frankly, you sound a little paranoid.”

Twitter stops backing up political tweets


A tweet is no longer forever — at least, if you’re a politician!

Herrman was reacting to an announcement from Twitter that it was facilitating the search of its entire archive of tweets, something it pitched as being critical for allowing “brands to most effectively analyze Twitter data.” Toxic tweets will be easier to find.

That was on August 11. On August 21, Twitter did something different: It shut off access to its application program interface, or API, for a tool that archived politicians’ tweets. It had previously stopped access to the API for Politwoops, a site that archived American politicians’ tweets. Now, projects gathering tweets from politicians in 30 countries and the European parliament are similarly disconnected.

Twitter defended the Politwoops move by saying that tweeting would be “nerve-wracking – terrifying, even” if you couldn’t delete your old tweets. “[D]eleting a tweet is an expression of the user’s voice,” it continued.

“What elected politicians publicly say is a matter of public record,” Arjen El Fassed, director of the group that operated those systems wrote in a statement. “Even when tweets are deleted, it’s part of parliamentary history. … What politicians say in public should be available to anyone. This is not about typos but it is a unique insight on how messages from elected politicians can change without notice.” For politicians, there are other mortal risks.

I made a similar point after Politwoops closed. When then-New York Democratic Rep. Anthony Weiner accidentally tweeted a photo and deleted it, that’s substantially different than when you tweet a picture and delete it. When politicians try to whitewash their records by deleting past public positions, that’s different, too. Politwoops used the power of programming to ensure that such things weren’t missed.

But this latest move is more alarming. Among the 30 countries that will no longer have the benefit of an automated system for backing up tweets are countries like Turkey and Egypt, countries without particularly open governments and which could benefit from more political accountability. It also includes countries like the U.K. and Australia where democracy has a firm grip but in which accountability is no less important.

Twitter’s argument is fairly simple. If you delete a tweet, it should be gone. If you don’t delete it, it should be able to be surfaced. That makes sense for a company trying to sell a service to advertisers and cater to a user base of consumers. Respects privacy, but takes advantage of its increasingly substantial data pool to allow deep analysis.

But Twitter isn’t just a company that matches consumers and advertisers. It’s an integral part of real-time global communications, including communications from elected officials. The failure to set a different standard for different types of users — especially as candidates increasingly use Twitter as part of their political campaigns — is a disservice to the community that uses it. This is not a court of law in which a comment can be stricken from the record. It’s a public square with a hot mic.

Evading the NSA


Did the feds think ignoring the laws could go on indefinitely, without anyone pushing back?

WASHINGTON — Devoted customers of Apple products these days worry about whether the new iPhone 6 will bend in their jean pockets. The National Security Agency and the nation’s law enforcement agencies have a different concern: that the smartphone is the first of a post-Snowden generation of equipment that will disrupt their investigative abilities.

The phone encrypts emails, photos and contacts based on a complex mathematical algorithm that uses a code created by, and unique to, the phone’s user — and that Apple says it will not possess.

The result, the company is essentially saying, is that if Apple is sent a court order demanding that the contents of an iPhone 6 be provided to intelligence agencies or law enforcement, it will turn over gibberish, along with a note saying that to decode the phone’s emails, contacts and photos, investigators will have to break the code or get the code from the phone’s owner.

Breaking the code, according to an Apple technical guide, could take “more than 5 1/2 years to try all combinations of a six-character alphanumeric passcode with lowercase letters and numbers.” (Computer security experts question that figure, because Apple does not fully realize how quickly the N.S.A. supercomputers can crack codes.)

Already the new phone has led to an eruption from the director of the F.B.I., James B. Comey. At a news conference on Thursday devoted largely to combating terror threats from the Islamic State, Mr. Comey said, “What concerns me about this is companies marketing something expressly to allow people to hold themselves beyond the law.”

Israel’s NSA scandal

So that’s how it happened. The Israelis have been blackmailing gay Palestinians to force them to be informants (always the hallmark of a democratic nation, I say!):

WASHINGTON — IN Moscow this summer, while reporting a story for Wired magazine, I had the rare opportunity to hang out for three days with Edward J. Snowden. It gave me a chance to get a deeper understanding of who he is and why, as a National Security Agency contractor, he took the momentous step of leaking hundreds of thousands of classified documents.

Among his most shocking discoveries, he told me, was the fact that the N.S.A. was routinely passing along the private communications of Americans to a large and very secretive Israeli military organization known as Unit 8200. This transfer of intercepts, he said, included the contents of the communications as well as metadata such as who was calling whom.

Typically, when such sensitive information is transferred to another country, it would first be “minimized,” meaning that names and other personally identifiable information would be removed. But when sharing with Israel, the N.S.A. evidently did not ensure that the data was modified in this way.

Mr. Snowden stressed that the transfer of intercepts to Israel contained the communications — email as well as phone calls — of countless Arab- and Palestinian-Americans whose relatives in Israel and the Palestinian territories could become targets based on the communications. “I think that’s amazing,” he told me. “It’s one of the biggest abuses we’ve seen.”

It appears that Mr. Snowden’s fears were warranted. Last week, 43 veterans of Unit 8200 — many still serving in the reserves — accused the organization of startling abuses. In a letter to their commanders, to Prime Minister Benjamin Netanyahu and to the head of the Israeli army, they charged that Israel used information collected against innocent Palestinians for “political persecution.” In testimonies and interviews given to the media, they specified that data were gathered on Palestinians’ sexual orientations, infidelities, money problems, family medical conditions and other private matters that could be used to coerce Palestinians into becoming collaborators or create divisions in their society.
Continue Reading →

Fewer uninsured patients benefit hospitals…

Who’d a thunk? Bloomberg is reporting that the drop in the number of uninsured patients has benefited hospital’s bottom line and has helped to slow down the rise of cost of healthcare…

HCA Holdings Inc. (HCA), the largest for-profit hospital chain, yesterday raised its forecast and reported a 6.6 percent drop in uninsured patients at its 165 hospitals, a reduction that grows to 48 percent in four states that expanded Medicaid, a top initiative of the Patient Protection and Affordable Care Act. WellPoint Inc. (WLP), which made the biggest commitment of any publicly traded insurer to the Obamacare markets, raised its guidance today after handily beating analyst estimates for the quarter on rising membership linked to the overhaul…

About 8 million Americans signed up for private plans offered through the health law’s insurance exchanges by April, and another 6 million were added to Medicaid, the state-federal program for low-income people, according to the Obama administration.

The proportion of the U.S. population without insurance has fallen 3.7 percentage points to 13.4 percent since the end of the 2013, according to Gallup Inc., the lowest rate since the firm began surveys of coverage in 2008.

“We’re now halfway through the first year of expanded coverage under the Affordable Care Act and, so far, our experience has been very positive,” William Carpenter, LifePoint’s chairman and chief executive officer, said in a July 25 conference call. The company operates 100 hospitals, according to data compiled by Bloomberg.

Medicare has been strengthened as well…

Medicare’s financial stability has been strengthened by the Affordable Care Act and other forces that have been subduing health-care spending, according to a new official forecast that says the fund covering the program’s hospital costs will remain solvent until 2030 — four years later than expected a year ago…

The trustees’ forecast said that the trust fund that pays for hospital care — Medicare Part A — has been strengthened significantly, with the date when it is predicted to start running short of money extended by 14 years since the Affordable Care Act was enacted in 2010. The report also predicted that the insurance premiums that older Americans pay for the portion of Medicare that covers doctors’ visits and other outpatient care would probably remain the same for a third year in a row.


This message may be recorded for quality purposes

Or for use by an algorithm that analyzes your voice and tells the customer service rep what to say to calm you down:

We already know insurers, United and Blue Cross are buying our MasterCard and Visa records.  It was odd and you can’t make this up, but Blue Cross said they buy them to look at to see if their insured members are starting to buy a size larger clothes.  You have to laugh at that we know they are querying a lot more than that as that’s what query masters do, it’s habit as I used to be a query master and your brain is stuck on queries to find more value.  It’s the way it works when you are developing software and of course once the SQL statements are done and given to
management, well they do all kinds of “scoring” as the next move as it will be tied to money.

It’s too bad they can’t get their claims processing working any better but they want to predict everything they can to include a mortality rate on you, and yes they do that too.  Remember though that predictions are based on patterns and there are levels of errors that will show up.

Again every company may not be set up to use voice analytics but it’s out there and I expect more to jump on as everyone wants every stick of predictive data and patterns they can get their hands on.  This is partly why we have such a glut of software out there today, everybody wants to score and analyze you ever way they can, whether it be underwriting or shipping ads your way but the problem as I wrote below is that with a lot of what we have out there “we don’t work that way” and conforming to software to change us will create upheaval and more desire to get the radar in time as there’s only so much we can take and again too when data and analytics keep getting resold and re-queried, the error factors rise.  There’s no incentive for correcting flawed data for consumers either as banks and big corporations have an absolute free labor pool and that is us to fix errors as we get denied something or access along the way after they have made their billions in profit selling us.

I am speaking to a man on the phone – but he’s not the only one listening. As I talk, software is analyzing my voice, measuring the speed of my speech, and building a graph that shows how the conversation is going.I’m talking to Josh Feast, CEO of a company called Cogito in Boston. His algorithms work away while people talk, highlighting awkward pauses, tense tones of voice and one-sided conversations. Next time you call your insurer, bank or any other call center, a version of Cogito’s software called Dialog could be running in the background, helping the customer service agent deal with you. If you start to get upset or angry, the agent can see that and take action to soothe you. Cogito calls its service “digital intuition”. It is useful in call centers because it can give feedback on conversations in real time, says Feast. One day, a version of it could even be running on your smartphone, analyzing every call you make and helping you spot if you are depressed, for example (see “A phone that listens“).

A blue bar that measures how much each person is speaking fills up as I listen to Feast. The fact he is dominating a phone interview about his software is to be expected, but if a call center agent saw that they were speaking as much as he is, they might want to ask more open-ended questions and bring the customer into the conversation. The software also measures the dynamic range of the caller’s voice, and the speed at which they speak. High dynamism, when a caller’s voice contains a lot of variation in pitch and emphasis, could mean they are excited or angry, for example. Low dynamism, a monotonous flat tone, might indicate disinterest or boredom.

Site Meter