ETL to QE, Update 3, Progress on first Report to be Published
Date: 2023-10-04
See Discord Binding for project context
Progress on first Report to be Published
I want to be able to generate both a generalized report summarizing all the Discord Guilds I have scraped as well as guild specific reports. Today I created a list of the Queries and Data Visualizations I want to in the generalized report in which can be found in Queries Comparing Discord Guilds.
I need data sorted and in a specific format before I can fit it into plotly the Data Visualization software I am currently using inside a jupyter notebook. I have been tempted to resolve SQL queries then use python to synthesize the data into the order and format I need for plotly but instead I decided to do as much in SQL as I can. I decided to do as much in SQL as possible because it will later allow me to possibly choose a different programming language from python for my backend, such as JavaScript, Go, Java, or Rust.
I also made an interesting discovery today using Plotly to produce multiple lines on a line chart. Turns out you can simply have one a dataframe with one column for labels, one for the timestamp, and one for the whatever value I want. I just point plotly at the right column on the dataframe and I don't need to think about parsing the data in any way myself. For reference check out, How to use Pandas and Plotly together?