Weather data analysis and visualization – Big data tutorial Part 6/9 – SED example

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 – Part 6/9

Manipulating output data with the Linux SED command – SED example

This Kartograph tutorial uses JSON as dataformat, so I needed the same format for my own data that the tutorial uses – example:

[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0 ? {"Weather": "MURSKA SOBOTA RAKICAN SI", "ll": [16.2, 46.7], ? "1962": 931.0}]

And the resulting data from PIG is not compatible with it, as it have different markup, presented here:

((ARAD RO,46.1331,21.35),{((1882,742)),((1883,680)),((1884,656)),((1885,656)),((1886,770)),((1887,718)),((1888,467)),((1889,893)),((1890,570)),

?

92))})

((DEVA RO,45.8667,22.9),?

So I?ve used the SED Linux command to alter the resulting dataset from Pig

SED example

Sed command 1

sed 's/(([A-Z ]*),([0-9.]*),([0-9.]*))/[{"Weather": "1", "ll: [3, 2], /g' rain_orig.csv > rain_new.csv && cat rain_new.csv

This command checks for ?(any number of capital characters)?, followed by ?,? and ?(any number of 0-9 numbers or ?.?)? blocks of two separated by a ?,? so is valid for

?((ARAD RO,46.1331,21.35)?

and changes this string to

[{“Weather”: ?ARAD RO?, “ll”: [21.35, 46.1331]

The syntax:

sed 's/  /  /g' input > output

for each string of the input find:

regular expression / change it to something.

1, 2 in SED Command 1 is a link to the regular expressions defined in the first part, in our case, 1 refers to ARAD RO, which is encapsulated in ? ? marks

I parse rain_orig.csv, the output of PIG and output it to rain_new.csv

Sed command 2

sed 's/{((([0-9]*),/"1": /g' rain_new.csv > rain_new2.csv && cat rain_new2.csv

The followings are set up as the above, parsing for {((number, and changing it to ?number? ? valid for years

Sed command 3

sed 's/)),((([0-9]*),/, "1":/g' rain_new2.csv > rain_new3.csv && cat rain_new3.csv

While this one parses for )),((numbers), and changes it to numbers

So after running all the 3 SED commands we have the needed results in the format of

<b>[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0, "1883": 680.0, ?</b>

One manual task left is to properly close the encapsulation at the end of the JSON file by changing }, to }]

The JSON for the Map is available now, and I?ve saved it to the directory of the new map visualization example.