Category Archives: Media Analytics

A/B Tests, 2-tail vs 1-tail tests & reporting the variance.

An article form June 2014 on how Optimizely and others only do one-tail tests, which simplifies the conclusion.

Entertaining article, here are some take-aways:

  • Do a test that is two-tailed (to see if there is significance for the opposite as well)
    • “The short answer is that with a two-tailed test, you are testing for the possibility of an effect in two directions, both the positive and the negative. One-tailed tests, meanwhile, allow for the possibility of an effect in only one direction, while not accounting for an impact in the opposite direction.”
  • Do several tests (to test if same hypothesis arises)
  • Run Tests Longer (to get more variety in the users)

Article: http://blog.sumall.com/journal/optimizely-got-me-fired.html

Some of the the hacker news comments (from January 2016) are interesting too, one mentioning that showing the variation is essential:

“In my view, the issue is not one-tail vs two-tail tests, or sequential vs one-look tests at all. The issue is a failure to quantify uncertainty.

Optimizely (last time I looked), our old reports, and most other tools, all give you improvement as a single number. Unfortunately that’s BS. It’s simply a lie to say “Variation is 18% better than Control” unless you had facebook levels oftraffic. An honest statement will quantify the uncertainty: “Variation is between -4.5% and +36.4% better than Control”.

(..)
“We just report credible intervals. We find that to be the only honest choice.

https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2…
https://www.chrisstucchio.com/blog/2013/bayesian_analysis_co…
https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic…

– yummyfajitas

https://news.ycombinator.com/item?id=10872359

Another point to raise is the issue on the impact on the Long Time Value of the user too. So, yes we got a higher CTR in the test, but how is the impact on LTV, is it the right kind of users who are clicking on that button?

Links on mobile app tracking for marketing

The history of use of unique identifiers for iOS products, UDID, MAC, browser cookie, OpenUDID. From May 2012.

http://www.cultofmac.com/160248/what-the-hell-is-a-udid-and-why-is-apple-worried-about-them-feature/2/

Mobile App Tracking, a company that boasts clientele like Zynga and LivingSocial, has proposed device fingerprinting. This approach is rather scatterbrained, as it needs apps to collect various bits of data about your iPhone (location, IP address, iOS version, etc.) and combine them together into one unique identifier. Unfortunately, users cannot easily opt out of fingerprinting or reset their device fingerprint altogether. There’s plenty of room for discrepancies since all of these bits of data can be meshed together and lose their accuracy over time. Device fingerprinting also requires licensing fees, so don’t expect it to gain widespread adoption.

And the history continues, idFA (Apple’s ID For Advertising) , articel from August 2013

http://www.adexchanger.com/data-driven-thinking/want-to-make-mobile-advertising-work-for-you-get-off-the-cookie-diet/

Though,

The main drawback to dynamic device IDs is the inability to support cross-device and cross-channel advertising performance attribution.

There are a few industry players working to address the attribution problem by creating an advertiser-agnostic central clearinghouse, where consumers opt in to a universal ID that can be used for targeted advertising and to measure engagement. This approach suffers from a heavy reliance on consumer and market education, and perhaps unrealistic expectations for consumer adoption and industry standardization.

Until then, we have to go back to our roots in science and math and to refocus on users rather than devices to overcome the limitations of cookies in mobile. Applying statistical probabilities to user behaviors online enables advertisers to estimate the likelihood that a user of one device is the same user of any other device, including tablets, smartphones, digital TV, desktop or any other digital device.

http://www.drawbrid.ge/  seems to do this.

“IDFA is back” , April 2014: 

http://www.appsflyer.com/idfa-is-back/

Mobile analytics:

https://count.ly/

http://piwik.org/blog/2012/04/how-to-use-piwik-to-track-mobile-apps-activity-clicks-phones-errors-etc/

Other relevant articles,

RTB, desktop ad networks coming to mobile ad networks, August 2013

http://www.admonsters.com/blog/id-key-unlocking-mobile-tracking-cross-device-measurement-part-i

Big Brands on mobile advertising

Mobile, Mobile Everywhere – But Where Are The Brands?

Attribution

Google To Buy Marketing Attribution Pure Play Adometry

How are the Top 100 Retailers Scaling for Mobile?