Skip to content

DentroptyDaemon Monorepo

ddaemon-monorepo

A collection of ETL(Extract Transform Load) tools for taking social media and quantified self data and generating visualizations so people can better understand themselves and others.

Motivation

Why can't Google index all my data allowing me to search myself? The vision for Dentropy Daemon is to take all ones data into a single extendable API that the user and only the user has access and control over.

Guide to Bindings and Apps

  • Bindings are tools to get and transform data from different sources into a format that can be queried.
  • Apps are tools that can be used to interact with the data once the bindings have it in the correct format

Bindings

Apps

dentropydaemon-wiki/Dentropy Daemon/Archive/Monorepo/backlog

Feedback

  • Fix the documentation for my project
  • Polish
  • Additional Charts
  • Backend API
  • Tell a bit of a story through the data
  • User Stories

General Backlog

  • Reference design for backend API
  • Start thinking about caching

Epics

  • Reaction type queries
  • Multi team/user/topic select
  • Messages per hour
  • Create API Backend rather than proxying the elasticsearch queries
  • Modular Query Builder function (Which can be moved to the backend later)
  • Add team / user select programmatically to each query rather than hard coding it in the JSON
  • Add specific team, user, and topic to query programmatically
  • Get most reactions to a specific message
  • Reply queries
  • Per capita queries
  • Finish adding the elasticsearch error response function in all components

Tech Debt

  • Review names for all reducers
  • Skim through everything
  • Place better console.logs as comments for later debugging
  • Remove the unused components
  • Margin and CSS stuff

Delete Specific Index

source .env
curl -XDELETE -iL -u $ELASTIC_USER:$ELASTIC_PASS "$ELASTIC_NODE/discordguild*"

Discord Binding

  • How many users in a guild
  • Add scripts for dumping and indexing each separate index
  • Get the character count of the contents
  • Get the number of attachments, mentions, embeds etc
  • Parse out any domain names

DataViz Backlog

  • What about specific user and topic queries
  • A specific user is not going to work because we want to graph the different message types
  • What about their activity across channel's?
  • For this it requires doing an aggregation on a separate field
  • This is worth doing
  • User message types is not going to work
  • General search functionality
  • Replies Graph
  • Topic's across team
  • User
  • Datagrid that can list complete messages
  • Combine the data grid components?
  • Longest
  • Replies
  • Most Reaction's

Git Binding

Repo Errors

https://github.com/Loopring/lightcone https://github.com/nemtech/nem2-workshop-nem-applied-to-supply-chain https://github.com/kusamanetwork/kusama.network https://github.com/Abracadabra-money/wonderland-frontend https://github.com/curvefi/curve-vue https://github.com/0xProject/contracts https://github.com/XYOracleNetwork/xyo-solidity https://github.com/oceanprotocol/ocean https://github.com/oceanprotocol/squid-js https://github.com/oceanprotocol/keeper-contracts https://github.com/oceanprotocol/pleuston https://github.com/singnet/singnet

Keybase Binding

  • Parse Reactions

Reminders

Current Task

  • REREAD THE USER STORIES
  • Review all user stories for the list queries and create a list of tech debt
  • Gotta fix the way the reducers values are named for the Query Selector especially for the lists

Research Notes

Design

  • wiki.software.Catagories.Data Visualization
  • Do I want the graph itself to be stored in the context?
  • Do I want to store each part of the graph as a separate reducer and use a separate context?
  • Yes but that is over optimization at this point
  • Reactions and Replies
  • Do I want to do an aggregation for reactions to a message for slack?
  • First I would have to seach all replies, then I would have to do an aggregation on the specific messages replied to

Feedback

  • Query replies
  • Per Capita message types
  • URL's per person
  • Emoji's sent by specific user distribution
  • Emoji's received by specific user
  • Graph of who interacts via emoji's
  • Time based stuff
  • Search across teams
  • Who is across multiple teams
  • CRM Integration
  • Hashtag support
  • Completed
  • Who has not posted in X Channel

Project Management.ddaemon-webapp.User Stories

Features*

Bindings

  • keybase-binding-nodejs
  • Get the attachments
    • Import topic list
    • Check if topic has been exported
    • Export or import topic
    • Find all attachments and save to attachments folder
  • List group chats
  • Export group chat
  • TODO ALSO EXTRACT CHARACTER COUNT, URLS, AND WORD COUNT FOR EDITS
  • discord-binding
  • Document how to get token
  • Document what is missing in Paul Mullins / DiscordChatExporter Helper · GitLab
  • facebook-binding
  • reddit-binding
  • pinboard-binding

Issues

  • We want a generalized way to do the elasticsearch queries
  • Why not have a switch statement and for each graph require a specific set of key's
  • This way it will be easy to get the graph title managed here as well
  • I guess this requires a custom component with props, yes it does
  • I need to add a props section to this wiki
  • There is not title for the graph
  • Research vega title
  • I want to make the sidebar built into the graph render component

Steps to add a graph

  • Create component template
  • Requirements
  • Graph name:
  • Graph ID:
  • Add component to KeybaseRoot.js
  • Add graph radio button to QuerySelect.js
  • Add to switch statement in KeybaseRoot.js
  • Automatically select Topic and Team in the left sidebar

Nice to haves

  • List users that are accounted for in the other part of the pie chart

Elastic Keys

  • user = msg.sender.username
  • topic = msg.channel.topic_name.keyword
  • team = msg.channel.name
  • Message type = msg.content.type