Columbia Journalism project Workbench closing

 

Workbench (https://workbenchdata.com/) the data journalism platform with built-in training; a project of Columbia Journalism School will close Oct 15. The project started in 2018 and is offering interesting tools and examples for data journalist. Pierre Conti, Co-founder and CEO writes: “ we did not succeed in creating enough revenue to make Workbench sustainable. Funding for journalism technology is scarce, and we did not manage to attract enough paid users to build a healthy business”. That is a pity of course, but you can build your own workbench running from you own machine. See instructions: https://github.com/CJWorkbench/cjworkbench/wiki/Setting-up-a-development-environment .

Is Windows transforming into Linux?

My first machine was running MSDOS is operating system(OS). On top of it you run a simple word processor, spreadsheets and database  software. It worked but a graphical user interface(GUI) was still under development. Things got slightly better with the introduction of Windows; Windows 95 was the big take-off. And of course adding to the OS the Office software: Windows became the tool of preference. Although Windows software was installed on  90% of the PC's., it was generally no fun. 

For example, you could get a blue screen or a completely frozen screen. You had to restart and if you did not saved your work you could start all over. Then the was the continuous updating by Microsoft; sometimes in the middle of your work. In the middle of a training Bill Gates stepped in to make things better.  And finally after all the updates, you had to find out where you could changed settings, the designers had changed the position and placed them under new headings.

Balmer

Finding an alternative OS was in the beginning not easy and demanded skill and knowledge of Linux. But Linux adapted fast and developed  a complete OS with different tools. There was a cold war between Microsoft and Linux. Microsoft CEO Steve Ballmer described Linux at that time as “cancer ”. Although Linux was developing in the direction of a serious alternative for Microsoft., and with no cost: Linux is open source and therefore free. It was never accepted. Running and installing Ubuntu or Mint is not rocket science, and working on these machines is slightly different from a windows machine. It never convinced Microsoft die hearts.

Microsoft may have won the battle for desktops and office software, but the servers and network software is all Linux. So the problem is to enable Windows to talk to all the Linux machines. Finally Microsoft saw the light and start embracing Linux.

Azure, Microsoft successful cloud computing is running Linux. So Microsoft needed it own  Linux version. Secondly this Linux version should also work on Windows machines. Let’s have a look at the various options.

 Linux on Windows

1. Installing Linux on windows is called WSL- Windows Subsystem Linux.The WSL comes in three  steps. Microsoft first introduced a Linux terminal in a distribution of choice, for example Ubuntu. Later it introduced a GUI, so you can run linux apps wit GUI.WSL1 only Linux terminal. Here is the documentation of Microsoft or read my blog posting in 2016. 

2. Later Microsoft introduced WSL2  with a GUI  using Xserver. If you want to give it try, here is a nice video showing how to install. Running Linux inside Windows is one thing; but it works also the other way. From Linux you have access to Window files and programs. Type for example explorer.exe at the Ubuntu prompt and you have the file explorer show the directory you in.

3. The latest is WSLg the GUI for Linux in Windows and it makes it possible to run directly Linux graphical apps. But you need to registers for the insider program and install windows developer version. It is to be expected that in fall with a new windows update WSLg will be implemented. 

Microsoft Linux

Microsoft own Linux version-CBL(Common Based Linux-Mariner). To install Microsoft Linux you can download the distribution and transform it into an ISO. To the get the feeling run it in a virtual machine. It comes  not with desktop and  GUI. CBL-Mariner is the Linux kernel used for running other Linux distributions

What is next?

Yes the cold war is over and peace broke out between Microsoft and Linux. Microsoft is definitely embracing Linux and incorporating it in windows. And also Microsoft is heavily involved in the development of open source software since it acquired Github. But it is not to be expected that Windows will be replaced by Linux. There is too much invested in other related Windows software to start a transition like this. But as a user you could run smoothly Linux apps.

However, now I have a Microsoft Linux distribution (CBL-Mariner) and I can run Linux apps graphical in Windows, and start Windows apps from the Linux prompt. Nice but so what…. ? I am not a developer. And as  user I prefer to run either Linux app or Windows apps, avoiding not the get mixed up. At the end of the day it is still running Linux in Windows in container. For the moment I stick to Linux(Ubuntu or Mint), which both give the ultimate control over the machines and the installed software.

                                                         

DATA JOURNALISM 2.0

Version 1.0

Analyzing and visualizing a table with figures for an article in a newspaper is not exceptional anymore. Take for example local taxes per municipality. Once you have downloaded the figures in a spreadsheet is not so difficult to to notice which of the local taxes generates the highest income for municipalities. A simple bar graph will do; or platting the tax income per municipality on a map will draw attention to the most expensive municipalities. Her are two examples made by Flourish about taxation for garbage collection in municipalities in the province of Gelderland. A bar graph: https://public.flourish.studio/visualisation/5152134/ and a map: https://public.flourish.studio/visualisation/5152371/ . Her is a dashboard made in Tableau: https://public.tableau.com/app/profile/verweijpjc/viz/reiniging/Dashboard1

Sometimes the data provider makes downloading data easy (http://d3-media.blogspot.com/2021/04/r-data-journalism-helpers.html ), bit often also provides analysis and visualisation(https://www.cbs.nl/nl-nl/nieuws/2021/04/gemeenten-begroten-11-3-miljard-euro-aan-heffingen-in-2021 ).

Here is the reporting of a local/regional newspaper about the taxes: https://www.gelderlander.nl/arnhem-e-o/kaart-deze-gemeenten-zijn-het-duurst-om-in-te-wonen~a217736c/

This is data journalism 1.0 and it is not the rocket science anymore it looked like 25 years ago. Data journalism 1.0 is almost 25 years old and the original beta version .0 is more tha 50 year old.(http://d3-media.blogspot.com/2018/07/new-steps-in-data-journalism.html ) A lot has been changed, most important is the tools are more easy to handle and require less skill and training. Data from a web page for example can directly imported in excel for example and visualizing based on a template from Flourish for example is almost standard procedure for journalists.

Version 2.0

On the other hand data software is developing fast. Data science is creating more software for analyzing data. Interesting for application in journalism is software generally known as Artificial Intelligence(AI) or Machine Learning(ML). You don’t need special machines for running those software, nor do you have to pay huge amounts for the use. A good laptop with a fast processor enough memory and a nice video device will do to run the software,which comes in two favors: R or Python. Skipping the discussion which is best, the difference in general is that both are capable of analyzing data; Python is a full programming language and R is more focused on statistics. I work mostly with R (http://d3-media.blogspot.com/2014/04/vijf-redenen-om-r-te-gebruiken-in-data.html ), running on a Linux operating system(OS)(http://d3-media.blogspot.com/2011/09/linux-voor-journalisten.html ). But Windows or Apple will also work.

R has a steep learning curve. Here is a howto start: http://d3-media.blogspot.com/2019/03/learning-r-for-data-journalists.html There is no graphical user interface(GUI), so you work from a terminal typing in commands or merge a set of command  into a small program. R has libraries for special job, and one set of libraries is dedicated to ML AI and or ML has lots of application.

Here is an example analyzing a dateset of municipalities in NL . I will use this data set also for explaining Machine learning. This example shows a standard analysis of the data in R;

https://www.kaggle.com/peterverweij/gemeente-test

This example is shown in Kaggle; more about this interface: http://d3-media.blogspot.com/2019/02/data-journalists-what-do-you-know-about.html

Machine Learning

ML has for data journalism has various area’s for implementation or application:

Automated content production or robot journalism (https://memeburn.com/2014/03/what-a-californian-earthquake-can-teach-us-about-the-future-of-journalism/ ) is one of them which drawing at the moment much attention. Another area is content optimalization: optimizing the content for a specific user.

For data journalism the second area data mining is the most interesting. The following chart gives an overview of the possibilities:

((chart from: https://nl.mathworks.com/discovery/machine-learning.html )

I will not discuss all these programs in detail.

First I will discuss the basic idea of machine learning and second I will show some examples. On data set based on data about dutch municipalities I will use to show: linear regression; decision trees and neural networks. An other dataset based on twitter I will use for showing K means.

The core of AI or ML is a black box: you give the box data input, next the box starts doing complicated statistical operations (the algorithm), which you fine tune with various option, and finally you have the output. Take for example the Titanic, there is complete data set about the passengers. With ML it is possible to calculate your changes to survive the shipwreck. Or to predict whether the mayor of a dutch municipality will be male or female.

For using the Ml you don’t need exact knowledge about the black box, the algorithm itself. That is for the coders or data scientist how design this software. The basic question for your research is what do you want to do?

From the chart we see that there are two entries: supervised and unsupervised learning. Supervised learning means that model(algorithm) which predicts the gender of the mayor of the changes on survival, must be trained on a known data set. When the model is trained, and you know then the margins of error, it can be apllied to complete data set to the predictions.

In unsupervised learning the data is immediately read into the an algorithm which makes the best of it. For example, analyzing tweets from two populist members of the Dutch house of representatives (Wilders and Baudet) show that their tweets have different clusters. Meaning they are both right wing populist but focus on different issues. Creating the clusters is a mathematical operation with no control data.

Under supervised learning we have to areas: classification and regression.

Unsupervised learning focuses on clustering.

Examples of machine learning

0. Kaggle

I will use Google Kaggle interface to show the code of the various algorithms. Here is a general intro to kaggle.

- machine learning in kaggle: http://d3-media.blogspot.com/2019/03/kaggle-is-there-data-journalism-in.html

1. Regression means that in a simple case with two variables, the variation of one variable is related to the variation and the other variable. Average income in a city for example will relate to the average price of houses; the relative rich cities houses will be more expensive. This relationship between the variables based on the variation they have in common can be expressed in a number: correlation. Or by a line a in scatter diagram, that is linear model, commonly called the trend(line)

When the number of variables increases the prediction of the outcome of one variable becomes more complicated. An the we have to use an other algorithm or model for the prediction. With decision trees it is possible to predict the gender of a mayor based om population, income, house price and unemployment; or predict the political party of the mayor based on the same variables. Here the predicted outcome is a category, nominal level of measurement

Neural networks make it possible to estimate the value, the quantity of a variable, ratio level of measurement. For example an estimation of the income of city, based on the other variables.

- regression: decision trees - random forrest: https://www.kaggle.com/peterverweij/prediction-simple-machine-learning

- regression: decision trees with rpart: https://www.kaggle.com/peterverweij/gender-rpart-predict-gem

- regression: neural networks with tensor flow: https://www.kaggle.com/peterverweij/kernel-tensorflow-woz

2. Clustering

The goal of clustering is to group or cluster observations that have similar characteristics, This is an example of unsupervised learning, so we have no control or check. Inspection of the output is the test. In the example below we regroup municipalities.

- clustering: https://www.kaggle.com/peterverweij/clustering-gemeentedata-using-kmeans

3. Classification- classification with voyant using nearest neighbor :

http://d3-media.blogspot.com/2020/03/two-faces-of-twitter-populism-in.html

Literature

More background and detail on machine learning for data journalists:

https://towardsdatascience.com/10-machine-learning-methods-that-every-data-scientist-should-know-3cc96e0eeee9

and

https://www.analyticsvidhya.com/blog/2017/09/common-machine-learning-algorithms/ )

and

http://d3-media.blogspot.com/2019/07/journalism-as-algorithm.html review of Automating the News. How Algorithms Are Rewriting the Media. By Nicholas Diakopoulos

1. ISBN 9780674976986 ; Harvard University Press 2019

R Data Journalism Helpers

R is an interesting platform that has found its way into data journalism. Once installed it is ready to use. However R is open source and is constantly developing. New packages are released, new ideas for analysis are discussed, for example at https://www.r-bloggers.com/ .

1. Get your data with an R api

R is a very important statistical platform for data scientist. Also in data journalism R has found applications and is used for example by journalists of the Economist or the BBC. (http://d3-media.blogspot.com/2019/02/blog-post.html ). Getting your data into R from let’s say a database puts you on a kind of detour. You make a selection of the data and download the set as .csv or .xls, from there you can import your data into R.

Direct access and downloading the data in R is now often an option. Database like the World Bank offer an R package called ‘wbstats’ ; more https://econandrew.github.io/wdi-api-R-gettingstarted/using-r-to-access-the-world-bank-api.html ; the IMF use ‘imfr’; more https://meshry.com/blog/downloading-data-from-the-imf-api-using-r/ ). For poverty data the World Bank uses ‘povcalnetR’; more https://github.com/worldbank/povcalnetR ). This is a great help for not only for data scientists but also journalist who are reporting for example about poverty and development in Sub Sahara Africa.

In the Netherlands, the Central Bureau of Statistics, has an interesting R api to collect data: ‘cbsodataR’. Here is an manual: https://www.cbs.nl/nl-nl/onze-diensten/open-data/statline-als-open-data/snelstartgids , For making maps the have the following handout: https://www.cbs.nl/nl-nl/onze-diensten/open-data/statline-als-open-data/cartografie

2. Scrapping and printing in R

Scrapping data from a webpage is generally not so difficult when employing Excel for analysis. For example if there is atable on the webpage, you could directly import the table in excel, using data and then from the web; paste the url and done. Outwit Hub is a nice programme for scraping.Table capture in chrome is an other posibility. Capture the dta in .csv and then import in Excel or in R.

Direct scraping data in R is more difficult, using package ‘rvest’, you have to dive deep into the .html code . More: https://www.dataquest.io/blog/web-scraping-in-r-rvest/

Now there is a package in R for scarping using a GUI calles ‘datapasta’; scraping a table from the web using R becomes a piece of cake. Here is a tutorial: https://milesmcbain.github.io/datapasta/articles/how-to-datapasta.html 

Making a plot with ggplot can also be time consuming because of the wide range on possibilities. A GUI for printing with ggplot is very helpful. I use a package called ‘esquisse’. Here are my experiences: http://d3-media.blogspot.com/2019/02/gui-for-r-ggplot.html

Data Journalism on the Samsung Tablet with Termux, Ubuntu and R

Can I turn my Tablet into a real work station? Apps are nice for small jobs, reading mail and bit of browsing. But for work…Office is a nice app and gives you all the standard Office tools like Word and Excel, lite versions of course, but working OK. But how about doing statistics with R? No R for Android so you need Linux environment. Installing Linux in a box in Android is possible with for example Linux Deploy app. However you need to root the machine. Generally I have no problem with that: I want full control on all machines. Rooting a machine has a consequence: losing the warrant. An secondly is a working root for this specific machine with that Android version available. And finally you run the risk in case of failure to turn you machine in a brick.

Data journalism on Samsung Tablet (1)

I had to buy a new tablet. The old one a Samsung Tab S10, SM-T800, was regularly crashing when it got too much data to load. Reading the NYTimes was a disaster, constantly reloading the pages; visalizations of the Economist crashed regularly because it was out of memory. The choice for a new one was: Samsung Tab 7+, SM T-970, running Android 11. Of course with a keyboard cover, just as the old one. Here they are: 

Adverteren in the politiek

 Sociale media hebben de politieke discurs fundamenteel veranderd. Het klassieke communicatiemodel met de journalistiek in het midden als gate keeper heeft plaats gemaakt voor direct communicatie tussen politicus en burger. Twitter en Facebook zijn de twee belangrijkste  sociale media en hebben beide een belangrijke politieke invloed. Via Twitter is bijvoorbeeld te beschrijven hoeveel invloed een twitterende policus heeft aan de hand van het aantal followers/friends. De inhoud van de tweets zegt iets over de standpunten. Eerder heb ik aandacht geschonken aan de tweets van Baudet en Wilders. Facebook pagina's en de volgers geven ook een indruk van de aanhang en standpunten van een partij.

Politieke advertenties op sociale media is een nieuw en groeiend fenomeen. Onlangs publiceerde NRC Handelsblad een insteressante analyse over campagne voeren op Facebook door Trump and Biden. Het artikel is gebaseerd op een analyse van data uit de advertentiedatabase van Facebook. In deze database is het mogelijk te zoeken op: 'issues, elections and politics' per land. Voor Nederland vinden we deze data base bijvoorbeeld voor de VVD gegevens over het totale bedrag dat aan advertenties is uitgegeven. Ook blijkt dat FVD veel geld investeert in dit soort advertenties en de PVV bijna niets.

De verkiezingen zijn pas volgend jaar, maar dit is toch een interessante database om verder te onderzoeken.