How To Parse JSON Data To A Human Readable Format [WIP]
from irmadlad@lemmy.world to selfhosted@lemmy.world on 09 Jun 17:27
https://lemmy.world/post/47961985

So, yes this is selfhosting related. I am working on an n8n flow to pull in weather data so that I can have this data on a dashboard. I can’t find any dockerized weather forecasting apps. Most of them connect to a personal weather station, which might be an option in the future. For the time being, this is a little project I’m working on.

Partial JSON snippet:

spoiler

0 json cod “200” message 0 cnt 40 list 0 dt 1780693200 main temp 29.4 feels_like 29.23 temp_min 29.4 temp_max 29.68 pressure 1019 sea_level 1019 grnd_level 984 humidity 42 temp_kf -0.28 weather

I would like to display something like this:

spoiler

Current temperature: 23.25 °C Feels like: 24.09 °C Low / High: 23.25 °C / 23.60 °C Humidity: 94% Atmospheric pressure: 1023 hPa (sea level: 1023 hPa, ground level: 988 hPa) Temperature correction factor (temp_kf): -0.35 Weather: Light rain Weather code: 500 Short condition: Rain Icon: 10d

So, this is for you devs or coders out there. I can produce the JSON data. I’m just not sure how to parse it to something meaningful. I’m sure Python will have to be incorporated, but unsure of how to proceed.

Maybe someone could point me in a direction to tuts, articles, or your own experience. Sorry the JSON data doesn’t format correctly. Lemmy formatting doesn’t seem to allow that.

#selfhosted

threaded - newest

mhzawadi@lemmy.horwood.cloud on 09 Jun 17:39 next collapse

Have a look at jq, a shell tool that can pass json and output in any way you want.

Or yes a python script will do that too

irmadlad@lemmy.world on 09 Jun 17:47 collapse

jqlang.org ?

Example tutorial: How To Transform JSON Data with jq

Thanks for the lead. I’ll see what I can do with jq.

non_burglar@lemmy.world on 09 Jun 22:12 collapse

In my dealings with json output, jq gets the job done.

verw@lemmy.dbzer0.com on 09 Jun 18:17 next collapse

First of all, there might be more out-of-the-box tools out there for building dashboards that require little or no programming knowledge, but I’ve never used or looked for them. Microsoft’s Power BI is one that I’ve heard of in passing, but fuck micro$lop lol. It’s just the only example I can think of that might be more approachable. You could google something like, “open source power BI alternative” to find other options.

Here are a couple of tutorials about using Python to parse JSON:

That will only cover how to interact with the JSON data, but not how to build a dashboard, ofc. Displaying the data in some fashion (e.g. a web page) is a whole other set of tasks.

If you’re new to programming in general, I recommend going through a collection of beginner tutorials on a programming language (Python is a good choice, but pretty much any will work for learning the basics). Any course like that will cover principles that apply to most languages and that would help you accomplish this task. For example, they’ll cover things like reading to and from files, parsing the content of files, and doing shit with the content.

Here are a few Python courses/tutorial collections you can check out. You don’t need to do all of them, but I wanted to give you some options so you’re more likely to find one that you vibe with:

EDIT: Btw, you are definitly not limited to using Python for this. Lot’s of other programming language will do. You just mentioned it, so I ran with that, haha.

irmadlad@lemmy.world on 09 Jun 18:40 collapse

If you’re new to programming in general,

I’m not new to programming, I’m just not very strong or proficient at all at it. You’ve given me a lot to read, and I thank you.

Btw, you are definitly not limited to using Python for this

I mentioned Python because that seems like a language a lot of people use for this sort of thing, but I am willing to entertain any other avenues…except perhaps Microsoft’s Power BI, which is discontinued.

Thank you.

curbstickle@anarchist.nexus on 09 Jun 19:00 collapse

Its not discontinued, btw, just now in the cloud and not free.

Still would not recommend though, its awful imo.

possiblylinux127@lemmy.zip on 09 Jun 18:49 next collapse

There are many different ways to do this

What is the usecase?

irmadlad@lemmy.world on 09 Jun 19:19 collapse

Use Case: I want to display a daily and extended forecast on a dashboard perhaps via an iframe or other means. I get tired of going to the weather channel and having them complain about my ad blocking techniques. LOL Also, I thought it would be a good learning exercise for me. I’m using n8n to pull the raw data from OpenWeather API. I can generate the JSON data fine (at least it was an accomplishment for me), it’s just getting that JSON data into a prettier format to display on a dashboard with maybe some pretty little weather icons later on when I get it down pat.

I mentioned Python because it seemed like a language I see others use for this sort of thing, but I am willing to explore other avenues. I am not very strong at coding. I can muck by, fiddling with this or that until I finally get something that works. Then I immediately commit that to my notes because my brain is shit at remembering things.

possiblylinux127@lemmy.zip on 09 Jun 19:56 next collapse

What dashboard are you using? It seems weird to use n8n as the data should already be in json format

Just FYI the data you provided is not in json format

irmadlad@lemmy.world on 09 Jun 20:27 collapse

I’m using Homarr which has the capabilities of iframes.

Is this not JSON format?

<img alt="" src="https://lemmy.world/pictrs/image/3a955605-3a7b-4b46-9ece-c8b230857eb3.png">

ETA: I guess I should have posted it as such:

spoiler

[{“json”:{“cod”:“200”,“message”:0,“cnt”:40,“list”:[{“dt”:1780693200,“main”:{“temp”:29.4,“feels_like”:29.23,“temp_min”:29.4,“temp_max”:29.68,“pressure”:1019,“sea_level”:1019,“grnd_level”:984,“humidity”:42,“temp_kf”:-0.28},“weather”:[{“id”:803,“main”:“Clouds”,“description”:“broken clouds”,“icon”:“04d”}],“clouds”:{“all”:65},“wind”:{“speed”:4.11,“deg”:213,“gust”:4.12},“visibility”:10000,“pop”:0,“sys”:{“pod”:“d”},“dt_txt”:“2026-06-05 21:00:00”},{“dt”:1780704000,“main”:

possiblylinux127@lemmy.zip on 09 Jun 22:11 collapse

Is there are reason you can’t just use the Homar weather plugin?

irmadlad@lemmy.world on 09 Jun 23:24 collapse

Well, primarily because I wouldn’t learn anything. lol Plus, the Homarr weather plugin doesn’t give as much detail as I would like. It works, no doubt, I just wanted something a bit more.

possiblylinux127@lemmy.zip on 10 Jun 02:00 collapse

You could build a weather app using flask and bootstrap 5. (Flask is the backend and bootstrap is the frontend)

obdurant@lemmy.zip on 09 Jun 21:56 collapse

I spent some time writing a script first in Python and then in ExcelScript at work to pull data from from OpenWeather API to compare ambient temperature and humidity to some of our indoor spaces. Both Python and JS (which ExcelScript is, basically) will make it very easy to pull the JSON from the API and pass whatever data you want down the line.

I don’t have any experience with dashboards or your setup in particular so I can’t give any good advice about how to go forward though. If you have a pre-existing dashboard, what makes it and what feeds it data? Is it a webpage or an app or such?

irmadlad@lemmy.world on 09 Jun 23:25 collapse

Is it a webpage or an app or such?

I am using the Homarr dashboard. It does have an iframe module that I think I could use once I pretty up the data.

obdurant@lemmy.zip on 10 Jun 02:54 collapse

Okay, I see what you’re going for and from another comment that you want to avoid the built-in weather app that dashboard has. I agree that the iframe should work for what you want.

In this case I would make a webpage you can host locally, ignoring the dashboard aspect for a moment, and have it format the data how you want it to show up. If you want to use this as an excuse to tinker with Python and learn a new project I would probably follow the tutorial for Flask on setting up a simple webapp. I imagine by the end of it you could use the example project and tweak it somewhat painlessly to render what you want. Since you already have the ability to get the weather JSON you’ll just need to link the two up.

Anyway, once you have the page working properly you’ll have something to put in the iframe.

curbstickle@anarchist.nexus on 09 Jun 18:58 next collapse

You’ve already got some options for the json, so I’ll just add:

dockerized weather forecasting apps

As a thought, maybe magic mirror? I’m (ab)using magic mirror as a dakboard/skylight style calendar, no mirror to be found, and the weather widgets are pretty good too. Another option would be Home Assistant if you’ve already got it up and running, you can set kiosk style with a single page to a specific logged in user, same with grafana if you’ve got it up. The Infinity data source is great for json ingest.

irmadlad@lemmy.world on 09 Jun 20:31 collapse

Home Assistant

I thought about Home Assistant, but I’m not sure what I’d do with the rest of it’s capabilities. I live in a house built in the latter part of the 40’s sitting in the middle of 22 acres. Besides a monitor for all my security cams which I already have set up, there’s not a whole lot to automate with Home Assistant around here. LOL 2 rooms, one bath, part of which have the original logs from when it was a log cabin. Not a whole lot to keep up with.

curbstickle@anarchist.nexus on 09 Jun 20:39 collapse

Oh see for me that sounds like a dream to set up HA for…. (also the kind of acreage I’m trying to get). Off the top:

  • Solar, track the energy generation and battery storage/health
  • Have outbuildings? Check on them with esphome sensors!
  • Into astronomy? TRACK IT THERE!

It mostly depends on what you’re doing with the property and what your goals might be. HA would be a lot to set up just for weather - magicmirror would definitely be the better choice for simple - but if you’ve got other stuff you are doing or want to do, HA would be a dream.

irmadlad@lemmy.world on 09 Jun 20:50 collapse

It mostly depends on what you’re doing with the property

Well, I used to be a farmer among other things in a former life. I’m 71 now and a bit too old to be running a farm full time. So I lease most of it out to other farmers in the area to grow silage for their livestock. I still retain some for my livestock, and personal grows. I also have parceled out a couple of half and quarter acre plots for the townies to grow their community gardens. Positive cash flow and all that.

curbstickle@anarchist.nexus on 09 Jun 21:13 collapse

So I don’t have the acreage right now, so I’m going to call this gardening level - but what I do with HA includes things like:

  • Soil monitoring, mostly moisture level but I’m experimenting a bit with some NPK sensors. I have a couple of solar battery banks that power them, brings all that back wireless.
  • Automated watering - rain barrel collection (with a sensor so I know how much is in there and so I don’t run a pump dry), relay control to open up the line and auto cut off. Combines a bit with weather monitoring so I don’t water heavily if rain is coming, waters in the evening to the morning to avoid evaporation
  • Flow sensor checks the water usage off the rain barrels so I know things are working right
  • Soil temp sensor is also on the list, not doing it right now just testing a few sensors still
  • Gate controls meant for a chicken coop, right now its just raising a flag for funsies, but when I get the right property it’ll close off the coop at night to protect the chickens
  • Light sensors to make sure the plants are getting enough, purely informational
  • Barometric sensors (originally just to be alerted before a storm comes through, though I like it now so I know when my knee is going to hurt like hell so I can prep)

Inside I have sensors to check for moisture around my water heater, automation for my lighting (like when I get out of my bed, the led strip turns on, brightness determined by room light levels and time of day), etc. I work from home, so I also have a light above my door that acts like an on-air light - if I join a call my wife/kids know not to interrupt me (light works great, kids still need operational adjustment here and there).

If all you want is that weather dashboard though….

Magic Mirror + OneCallWeather module + rain map module and you’ve got a slick setup, maybe add in some camera streaming via rtsp for a bonus. And if you go homeassistant later, you can add a module for that too.

irmadlad@lemmy.world on 09 Jun 23:25 collapse

so I’m going to call this gardening level

Micro Farming. ;)

curbstickle@anarchist.nexus on 09 Jun 23:28 collapse

I feel like thats giving me too much credit haha

I am really into the idea of doing some hydroponics and microgreens too though - thats what I’m prepping for this winter at least!

yes@lemmy.dbzer0.com on 09 Jun 20:35 next collapse

You can point the homarr iframe to a HTML file that consumes the n8n json output with JavaScript (fetch and transform data)

Even easier might be to point the iframe at the n8n endpoint but to add a HTML node as last node to your workflow. There you can use plain HTML and it directly supports the expression syntax to render the values from your previous json node.

chrash0@lemmy.world on 09 Jun 22:36 next collapse

i use Nushell for this! works with JSON, YAML, TOML, markdown, Polars Dataframes, SQLite, and a bunch of others including builtin parsing tools for whatever formats and a plugin ecosystem. i use it at work and for personal projects as my main shell, and it’s super handy for exploring, unpacking, sorting, and visualizing all sorts of data. i use it to:

  • find specific parts of YAML cloud configs
  • visualize JSON logs, including a parser that restructures journalctl logs.
  • _re_structure data from CLIs to work with them as structured: git logs, Unix coreutils, etc
  • script my environment: common kubectl queries, specific web API helpers, building and running and testing applications, etc

it is a slight learning curve, and technically you could do all of that with bash or zsh and jq or jc, but i appreciate the modern take on your base shell terminal env.

it’s replaced both Python and Bash for me.

irmadlad@lemmy.world on 09 Jun 23:21 collapse

Nushell

www.nushell.sh? I’ll add it to this evening’s reading material. I was skimming the site and saw this: www.nushell.sh/cookbook/jq_v_nushell.html in relation to jq.

Thanks for the input.

chrash0@lemmy.world on 10 Jun 00:48 collapse

exactly! i basically live in the terminal, and this is my go-to shell for all platforms

GreenKnight23@lemmy.world on 09 Jun 23:01 next collapse

lots of ways to do it, where will it be displayed? if it’s in a console, jq would more than suffice.

if you want it to be processed heavily and added to an HTML document, probably something like a mix of jq and envsubst. you could forego all that and just use sed to inject the json into HTML then have JavaScript parse it out for you browser side.

if you want to store it in a database why not just store the whole json?

your requirements aren’t very clear on what you want this data for or how it will be used. without that there’s just too many variable solutions.

can you parse json? sure! what for/how will it be used? not sure.

hope this helps.

irmadlad@lemmy.world on 09 Jun 23:18 collapse

envsubst sed

Ahh more to explore. Thanks.

your requirements aren’t very clear on what you want this data for or how it will be used.

Apologies. As per usual, it’s all in my head and sometimes doesn’t make it to paper. I am using Homarr dashboard. Homarr has the ability to incorporate iframes. Ideally I would like to parse the JSON data into a prettier format perhaps adding some icons in the future once I get it all hammered out, and display it in an iframe on my dashboard.

Thank you for the leads and your time.

GreenKnight23@lemmy.world on 09 Jun 23:31 collapse

I don’t know much about homarr but it sounds like going with jq and envsubst might give you a faster solution that you can just iframe in your dashboard. though the learning curve can be steep if you’re unfamiliar with Linux command line.

I’m pretty sure you can literally pass in base64 encoded HTML to an iframe it will load in or pass it a publicly accessible file and it will work too.

I had messed around with building a locally hosted dashboard using similar methods so I’m pretty sure it will work.

captcha_incorrect@lemmy.world on 10 Jun 00:11 next collapse

Since you are using n8n, pull the data on a schedule and save it in a data table. Then create a webhook to serve the content as you want it. N8n can transform the JSON data, I can provide a sample tomorrow.

irmadlad@lemmy.world on 10 Jun 13:17 collapse

I can provide a sample tomorrow.

That would be swell, but I wouldn’t make it a priority. Just whenever you get a round to it. I have been given a ton of source material to read. Stayed up a little too late last night trying to assimilate it all. LOL

rimu@piefed.social on 10 Jun 00:52 next collapse

In the Homarr docs, there is this: https://homarr.dev/docs/widgets/iframe/

Towards the end there is a link to https://github.com/diogovalentte/homarr-iframes which contains many examples of some content formatted nicely to fit within Homarr iframes and a docker container you can use to make a url that the iframe widget can consume. They used Go which is only a little harder than Python to read.

Seems like this one - https://github.com/diogovalentte/homarr-iframes/tree/main/src/sources/changedetectionio - makes a request to a JSON api, for example.

K3can@lemmy.radio on 10 Jun 09:57 next collapse

Personally, whenever I need to process anything text-based, I use perl.

Read the json into a hash, parse the values if desired, then plug the values into an html template.

It’s pretty quick to write, much easier to learn than python (in my opinion), and super powerful.

irmadlad@lemmy.world on 10 Jun 13:09 collapse

I know what perl is, but I have never used it before. I’ll spool up on that too.

justme@lemmy.dbzer0.com on 10 Jun 15:19 collapse

isn’t json already a human readable format? ;) ok, jokes aside. to parse a file interactively, like i need to extract some information now, i use Julia, because I’m working with it anyway. that’s of course not very practical for automation scripts. most convenient would be a language that is already incorporated in your setup. otherwise i usually end up with a bash script with some jq calls.