A piece of software that shouldn't need to exist. Processes badly formatted outputs from Morgan Stanley Shareworks into usable CSV files.

Shareworks-munger: A program to process Shareworks reports into Something Usable

"Shareworks" is a product of Morgan Stanley which tracks certain kinds of financial information (specifically, stock and compentation-related information) and distributes reports about it to the receiptients.

Inscruitably, it offers this in two formats: a PDF, and an HTML page. Both of them have tables in a format that will make your accountant absolutely scream -- they're not in rows; there'd be a ton of manual data re-entry; it's just no good. There is no CSV export, or anything you'd expect.

This is a program that takes the HTML form, parses it, puts it into a sane columnar layout, and emits the data in CSV.

That's it.

You can then open the CSV in Excel, or LibreOffice, or whatever you please.

Side by side, messy data and clean data

What the Fuck?

We're talking about the data exports from a piece of software that's A) by a major financial company, and B) literally has no purpose except data reporting...

and I'm telling you that their exports are too batshit for an accountant to actually use?

Yes. Yes I am.

I'm not kidding about turning dozens of pages into a few rows in the CSV form, either. It's really obscene.

I cannot begin to speculate on what kind of failure of organization inside Morgan Stanley caused them to utterly fail to be able to do something like "emit a CSV report", but fail they did.

This is not software that should've needed to be written.

None the less, it was needed; and so I have written it.

License

While I do hope you are not in the position of needing this, if you do, it is a free and open license.

More specifically: Apache2 or MIT licensed, at your option.

I really don't care what you do with it; I just want to increase the amount of sanity in the world, and this is a contribution.

How to Use

Requirements

  1. You still have to log into the Shareworks website manually, and navigate to the reports, and save that webpage from your browser.
  2. You'll need Git and a Golang compiler installed and some basic familiarity with working in the terminal. (I couldn't be arsed to package this any better. Sorry.)

In other words, this will require you to be somewhat technical, still.

If you'd like to make this more automated and easier to use, "PRs welcome", as we say in the tech industry.

Steps

Getting the raw data

  1. Log in to Shareworks.
  2. Get the report onto your screen.
    • Click "Activity" at the top.
    • Click "Reports".
    • Do the date selection to whatever you want.
    • Leave the other options as "Web Page" and "Full".
    • Submit.
  3. Now it gets fun. We need the HTML from this, but we can't just save the page.
    • (Why can't we just save the page? Because this website... it's incredible, in a bad way. I don't wanna talk about it. This is not how this website should've been written.)
    • Right-click somewhere in the report data, and then click "inspect".
      • Some crazy debugger thing just opened. If you don't know what this is, don't worry, just hang on.
    • Scroll up until you see something that says <iframe id="transaction-statement-iframe (... and then some more stuff you can ignore; it just has to start like this).
    • Scroll slightly back down: You should see a bit that says <html right underneath the <iframe. Right click on that.
    • Hover "Copy" in the right-click menu, then click "Inner HTML" -- or "Copy element" in Chrome. (Whatever.) You should now have a big hunk of juicy data on your clipboard! We're almost there!
    • Dizzy? See the screenshot below.
  4. Paste that data from your clipboard into a file and save it. Call it wow.html or whatever you want.
    • If you're not very tech savvy -- mind that this needs to be a "plain text" file. Not a word document or whatever. If that's not familar to you, I'm sorry; this is beyond my depth to explain in this readme.
  5. HOORAY -- you are done with the browser now. You can close it.

Here's a screenshot of what getting the raw HTML looks like:

How to copy the goddamn html

Okay. Phew. Now on to actually running the munger.

Using the Munger

  1. Git clone this repo, if you haven't already.
    • If you're not the kind of tech savvy for this -- I'm sorry; this is beyond my depth to explain in this readme.
  2. You should have Golang installed. Sanitycheck: you can run go env in the terminal, and it works, right?
    • If you're not the kind of tech savvy for this -- I'm sorry; this is beyond my depth to explain in this readme.
  3. go run ./main.go ./wow.html -- or use whatever your filename was from step 4 above, when you got the data.
  4. That's it! The CSV data should've appeared on your terminal!
  5. Redirect it to a file to save it: go run ./main.go ./wow.html > sane.csv

You should now be able to open sane.csv with Excel, or LibreOffice, or whatever you want! And you can go ahead and send it to your accountant; they won't hate you anymore. (Probably. At least not for this issue.)

Owner
Eric Myhre
hash *all the things!* Working on trusted computing and reproducible builds.
Eric Myhre
Similar Resources

SkillcappedRippr - Rips (downloads) videos from skilcapped without the need to be logged in to the website

skillcappedRippr Rips (downloads) videos from skilcapped without the need to be

Sep 14, 2022

✏️ CLI tool to split a file into smaller sub-files

filesplit CLI tool to split a file into smaller sub-files Build $ go build Usage filesplit [mode] [-F, --file] [-N, --number] Examples # Split foo.txt

Apr 20, 2022

A CLI tool which loads data from yaml files into the Google Cloud Spanner tables

splanter A CLI tool which loads data from yaml files into the Google Cloud Spanner tables (mainly for the development).

Oct 27, 2022

Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.

Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.

dasel Dasel (short for data-selector) allows you to query and modify data structures using selector strings. Comparable to jq / yq, but supports JSON,

Jan 2, 2023

A simple script to run speedtest(offical) CLI tool and store the results in CSV

PeriodicBW A script made to run official speedtest.net binary periodically and store the results in a CSV file Installation Get the official speedtest

Aug 10, 2021

CLI tool to rank proposals according to Majority Judgment, from an input CSV

CLI tool to rank proposals according to Majority Judgment, from an input CSV

Majority Judgment CLI tool WORK IN PROGRESS Although the core mechanics are here and ranking does work, the features described in this README are not

Sep 10, 2022

A simple command line for convert CSV in JSON

A simple command line for convert CSV in JSON

C2J A simple command line for convert CSV in JSON list of objects based on header. Install With Go 1.17 or higher: go install github.com/edermanoel94/

Dec 14, 2022

Command line tool for processing client transaction data in CSV format

command line tool for processing client transaction data in CSV format. thank you for looking! build instructions: $ git clone https://github.com/lnit

Oct 29, 2021

Smart and simple CSV processing on the command line

csvquote smart and simple CSV processing on the command line Are you looking for a way to process CSV data with standard UNIX shell commands? Are you

Dec 30, 2022
Limner colorizes and transforms CLI outputs.
Limner colorizes and transforms CLI outputs.

Limner Limner colorizes and transforms CLI outputs. Motivation When playing with kubectl, I sometimes found it hard to extract the information I neede

Dec 14, 2022
simple clipboard copy for complex outputs
simple clipboard copy for complex outputs

Terminal Clipboard Different from others terminal's clipboard coppiers, this one can copy complex and interminables commands like tail -f or outputs t

Nov 5, 2021
A simple CLI tool that outputs the history of connections to Amazon EC2 instances using AWS Session Manager.

ssmh This is a simple CLI tool that outputs the history of connections to Amazon EC2 instances using AWS Session Manager. Installation brew install mi

Dec 10, 2021
Go program that outputs a GIF with the lissajous figures using green and black

lissajousgb This is an exercise of the book The Go Programming Language, by Alan

Dec 18, 2021
Program to convert plain text to CSV file which can imported into Anki.

Program to convert plain text to CSV file which can imported into Anki. The motivation of this program is to save time by automatically coverting Question and Answer into CSV file which can be imported directly into Anki.

May 22, 2022
A fast diff tool for comparing csv files

csvdiff A fast diff tool for comparing csv files. What is csvdiff? Csvdiff is a difftool to compute changes between two csv files. It is not a traditi

Dec 28, 2022
Command-line tool to load csv and excel (xlsx) files and run sql commands
Command-line tool to load csv and excel (xlsx) files and run sql commands

csv-sql supports loading and saving results as CSV and XLSX files with data processing with SQLite compatible sql commands including joins.

Nov 2, 2022
Easy to use library and CLI utility to generate Go struct from CSV files.

csv2struct Easy to use library and CLI utility to generate Go struct from CSV files. As a benefit, it's fully compatible with csvutil. So, structs gen

Nov 7, 2022
Money Cheat - Need For Speed Underground 2

nfsu2-money-cheat Money cheat for "Need For Speed Underground 2" -- allows you to edit/change money within your NFSU2 save file. Download EXE Link: ht

Nov 3, 2022
A CLI tool for working with CloudWatch logs. It performs functions that I need at work.

CloudWatch Logs Utility A simple utility for working with CloudWatch Logs. AWS should probably build this themselves, but since they won't, I am here

Dec 31, 2021