Golang library for querying and parsing OFX

OFXGo

Go Report Card Build Status Coverage Status PkgGoDev

OFXGo is a library for querying OFX servers and/or parsing the responses. It also provides an example command-line client to demonstrate the use of the library.

Goals

The main purpose of this project is to provide a library to make it easier to query financial information with OFX from the comfort of Golang, without having to marshal/unmarshal to SGML or XML. The library does not intend to abstract away all of the details of the OFX specification, which would be difficult to do well. Instead, it exposes the OFX SGML/XML hierarchy as structs which mostly resemble it. Its primary goal is to enable the creation of other personal finance software in Go (as it was created to allow me to fetch OFX transactions for my own project, MoneyGo).

Because the OFX specification is rather... 'comprehensive,' it can be difficult for those unfamiliar with it to figure out where to start. To that end, I have created a sample command-line client which uses the library to do simple tasks (currently it does little more than list accounts and query for balances and transactions). My hope is that by studying its code, new users will be able to figure out how to use the library much faster than staring at the OFX specification (or this library's API documentation). The command-line client also serves as an easy way for me to test/debug the library with actual financial institutions, which frequently have 'quirks' in their implementations. The command-line client can be found in the cmd/ofx directory of this repository.

Library documentation

Documentation can be found with the go doc tool, or at https://pkg.go.dev/github.com/aclindsa/ofxgo

Example Usage

The following code snippet demonstrates how to use OFXGo to query and parse OFX code from a checking account, printing the balance and returned transactions:

client := ofxgo.BasicClient{} // Accept the default Client settings

// These values are specific to your bank
var query ofxgo.Request
query.URL = "https://secu.example.com/ofx"
query.Signon.Org = ofxgo.String("SECU")
query.Signon.Fid = ofxgo.String("1234")

// Set your username/password
query.Signon.UserID = ofxgo.String("username")
query.Signon.UserPass = ofxgo.String("hunter2")

uid, _ := ofxgo.RandomUID() // Handle error in real code
query.Bank = append(query.Bank, &ofxgo.StatementRequest{
	TrnUID: *uid,
	BankAcctFrom: ofxgo.BankAcct{
		BankID:   ofxgo.String("123456789"),   // Possibly your routing number
		AcctID:   ofxgo.String("00011122233"), // Possibly your account number
		AcctType: ofxgo.AcctTypeChecking,
	},
	Include: true, // Include transactions (instead of only balance information)
})

response, _ := client.Request(&query) // Handle error in real code

// Was there an OFX error while processing our request?
if response.Signon.Status.Code != 0 {
	meaning, _ := response.Signon.Status.CodeMeaning()
	fmt.Printf("Nonzero signon status (%d: %s) with message: %s\n", response.Signon.Status.Code, meaning, response.Signon.Status.Message)
	os.Exit(1)
}

if len(response.Bank) < 1 {
	fmt.Println("No banking messages received")
	os.Exit(1)
}

if stmt, ok := response.Bank[0].(*ofxgo.StatementResponse); ok {
	fmt.Printf("Balance: %s %s (as of %s)\n", stmt.BalAmt, stmt.CurDef, stmt.DtAsOf)
	fmt.Println("Transactions:")
	for _, tran := range stmt.BankTranList.Transactions {
		currency := stmt.CurDef
		if ok, _ := tran.Currency.Valid(); ok {
			currency = tran.Currency.CurSym
		}
		fmt.Printf("%s %-15s %-11s %s%s%s\n", tran.DtPosted, tran.TrnAmt.String()+" "+currency.String(), tran.TrnType, tran.Name, tran.Payee.Name, tran.Memo)
	}
}

Similarly, if you have an OFX file available locally, you can parse it directly:

func main() {
	f, err := os.Open("./transactions.qfx")
	if err != nil {
		fmt.Printf("can't open file: %v\n", err)
		return
	}
	defer f.Close()

	resp, err := ofxgo.ParseResponse(f)
	if err != nil {
		fmt.Printf("can't parse response: %v\n", err)
		return
	}

	// do something with resp (*ofxgo.Response)
}

Requirements

OFXGo requires go >= 1.12

Using the command-line client

To install the command-line client and test it out, you may do the following:

$ go get -v github.com/aclindsa/ofxgo/cmd/ofx && go install -v github.com/aclindsa/ofxgo/cmd/ofx

Once installed (at ~/go/bin/ofx by default, if you haven't set $GOPATH), the command's usage should help you to use it (./ofx --help for a listing of the available subcommands and their purposes, ./ofx subcommand --help for individual subcommand usage).

Owner
Comments
  • "Windows" new line required for some banks

    It looks like some banks, like USAA, don't support standard unix line endings \n:

    OFXAdapter: Failed to parse request: Unable to parse an atomic field "OFXRequest.Headers.OFXHeader": unable to find end delimiters "
    ".
    

    Looking around, I found this issue which solved the problem by adding \r: https://github.com/euforic/banking.js/issues/8

    Based on the code in the python ofxclient library, it looks like using \r\n works for all banks: https://github.com/captin411/ofxclient/blob/v2.0.4/ofxclient/client.py#L25

    In my custom client, I've hacked around the problem with this line:

    var r *ofxgo.Request
    b, _ := r.Marshal()
    b = bytes.NewBuffer(bytes.Replace(b.Bytes(), []byte{'\n'}, []byte{'\r', '\n'}, -1))
    

    What are your thoughts on changing the line ending for all requests? If you like I can try my hand at a PR too 😄

  • Added Accept http header as Citi now requires it

    Added Accept http header as Citi now requires it

    Without this header set Citi returns http 403 for every request.

    I spent far to long looking into why Citi started returning only 403s for me. I eventually found that the software pocketsense (https://pocketsense.blogspot.com/2021/08/dev-testing-continued.html) was having a similar issue and it was resolved by adding this http header to the requests. I confirmed it resolves the issue for me as well.

  • Rename ofxgo_test package to ofxgo and remove self-imports/references

    Rename ofxgo_test package to ofxgo and remove self-imports/references

    It turned out that removing the self-imports was more involved than I previously thought! This PR renames the test package to use the same name (ofxgo) so imports aren't required to run the tests. I've also un-exported ErrInvalid so it is no longer part of the API surface area.

    Luckily sed is amazing, so 99% of the renaming and import deletion effort was automatic 😅

    Since this touches so many lines in the library, I figured it'd be worth breaking this out into it's own PR, for our sanity.

  • can't parse response: OFX SECURITY header not NONE

    can't parse response: OFX SECURITY header not NONE

    Parsing OFX responses with SECURITY setting of TYPE1 fails with a validation error:

    can't parse response: OFX SECURITY header not NONE
    

    My bank still uses OFX 1.02 OFXSGML format, but uses SECURITY:TYPE1:

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:TYPE1
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    

    This is parsed beautifully after I change that security setting to NONE:

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:NONE
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    

    Apparently, this is a common issue in financial software processing .ofx responses, with the "fix" for users to hand-edit the responses to remove the TYPE1 security value and replace it with NONE:

    When I went digging to learn more about this setting I found Open Financial Exchange Specification 2.2, Nov 26, 2017 which states in section 4.2.2.2 Type 1 Protocol Overview that:

        Type 1 applies only to the request part of a message; the server response is unaffected.
    

    Thus I think we could remove [this validation](- https://github.com/aclindsa/ofxgo/blob/2b8a79e4b74b0ca32c2b6f21147841d22c656dca/response.go#L95-L98 ) as TYPE1 security doesn't seem to impact responses.

  • Return parsed response when only failing validation

    Return parsed response when only failing validation

    Hey again Aaron! Let me know your thoughts on this more flexible version of ofxgo.ParseResponse. It still returns an error if things are not parseable, but attempts to continue decoding if it's only hitting Valid()-ation errors. This way I can write institution-specific handling of validation errors and still use the response object.

    I ran into this issue with Ally Bank where their severity in the transaction list is not uppercase like the OFX 102 spec requires. I've contacted them, but I don't expect they'll actually change anything since "it works for Quicken". 😉

  • Marshalled requests for OFX 102 should not include element end tags

    Marshalled requests for OFX 102 should not include element end tags

    The OFX 102 spec is a bit lacking in this area, but it seems that some institutions require ofx elements not include their end tags (aggregates are ok though).

    The key rule of Open Financial Exchange syntax is that each tag is either an element or an aggregate. Data follows its element tag. An aggregate tag begins a compound tag sequence, which must end with a matching tag; for example, <AGGREGATE> ... </AGGREGATE>.

    From the reading above and my use of the python ofxclient, I think always omitting the element end tags should work for all institutions using OFX 102 (maybe all 100-series, but I haven't looked yet).

    In my case, USAA doesn't like me using any element end tags. This is my hacky work-around:

    var r *ofxgo.Request
    b, _ := r.Marshal()
    
    data := b.Bytes()
    for _, s := range []string{"DTCLIENT", "DTSTART", "DTEND", "INCLUDE", "USERID", "USERPASS", "LANGUAGE", "ORG", "FID", "APPID", "APPVER", "TRNUID", "BANKID", "ACCTID", "ACCTTYPE"} {
    	data = bytes.Replace(data, []byte("</"+s+">"), []byte{}, -1)
    }
    b = bytes.NewBuffer(data)
    

    What do you think would be a good solution to this problem? I briefly looked at your fork of go's xml package, looks like that might be a good place for it. Maybe it could be some kind of encoder option?

  • QFX downloads from BMO do not have a newline between SGML and XML section

    QFX downloads from BMO do not have a newline between SGML and XML section

    This is how the beginning of the file looks like

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:NONE
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    <OFX>
    <SIGNONMSGSRSV1>
    <SONRS>
    <STATUS>
    <CODE>0
    <SEVERITY>INFO
    <MESSAGE>OK
    </STATUS>
    ...
    

    I am not sure if this is technically valid OFX, but Bank of Montreal is a major Canadian Bank. Feel free to reject if you think the bank is plain wrong. I should also mention that the bank refers to the export format as QFX, which is possibly a variation of OFX.

  • BMO produces invalid XML for BANKMSGSETV1

    BMO produces invalid XML for BANKMSGSETV1

    Bank of Montreal (BMO) allows downloads of multiple accounts at once. It uses BANKMSGSETV1 element for that (with embedded BANKMSGSET inside it for some reason), but it emits start elements instead of end elements at the end 🤦‍♂️ .

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:NONE
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    
    <OFX>
    <SIGNONMSGSRSV1>
    <SONRS>
    <STATUS>
    <CODE>0
    <SEVERITY>INFO
    <MESSAGE>OK
    </STATUS>
    <DTSERVER>20181228000224.466[-5:EDT]
    <USERKEY>329402394029374
    <LANGUAGE>ENG
    <INTU.BID>00001
    </SONRS>
    </SIGNONMSGSRSV1>
    <BANKMSGSETV1><BANKMSGSET>
    <BANKMSGSRSV1>
    
    <STMTTRNRS>
    ...
    </STMTTRNRS>
    
    </BANKMSGSRSV1>
    <BANKMSGSET><BANKMSGSETV1>
    </OFX>
    

    It seems that the library currently doesn't support MSGSET elements at all, but even if it did the end elements would still be a problem. I could get the file through by making response parsing ignore MSGSET elements completely, as follows:

    @@ -356,7 +357,13 @@ func ParseResponse(reader io.Reader) (*Response, error) {
                            return nil, err
                    } else if ofxEnd, ok := tok.(xml.EndElement); ok && ofxEnd.Name.Local == "OFX" {
                            return &or, nil // found closing XML element, so we're done
    +               } else if ofxEnd, ok := tok.(xml.EndElement); ok && strings.Contains(ofxEnd.Name.Local, "MSGSET") {
    +                       continue // ignore MSGSET elements
                    } else if start, ok := tok.(xml.StartElement); ok {
    +                       // ignore MSGSET elements
    +                       if strings.Contains(start.Name.Local, "MSGSET") {
    +                               continue
    +                       }
                            slice, ok := messageSlices[start.Name.Local]
                            if !ok {
                                    return nil, errors.New("Invalid message set: " + start.Name.Local)
    

    Note that I had to ignore EndElements as well, to allow the parser to unwind its element stack properly. The way the sample above is parsed is as if there are two sets of MSGSET elements nested in each other and the end elements get injected from the parser stack when it encounters the closing OFX element.

    Anyway, I admit that this is outright disgusting, but it does seem to allow processing the file correctly including multiple account message sets. I absolutely understand if this is too much of a hack to allow in. I can muscle my way through outside of the ofxgo library if needed (just use some grep -v MSGSET equivalent), but I figured I'd bring it up anyway.

  • Pending transactions missing from query responses

    Pending transactions missing from query responses

    I'm currently using my personal Chase Bank checking, savings and credit accounts to play with this package. Most things are working as expected however I am not able to get my statement balances and transactions to be as up to date as I would like. It appears the responses ignore pending charges entirely regardless of whether the IncludePending flag is set to true. Have you seen this behavior before? I have to imagine retrieving pending charges would be supported by chase but I could be wrong.

  • slow go-get because of aclindsa/go dep

    slow go-get because of aclindsa/go dep

    Thanks for releasing this library.

    I'm excited to use it, but it's taking a long time to download. go get is getting stuck on downloading aclindsa/go, which seems to be a big repo.

    Is it possible to use the default Go encoding/xml or put a fork in a smaller git repo?

  • Unable to connect to any of my banks using this

    Unable to connect to any of my banks using this

    query.URL = "https://www.oasis.cfree.com/fip/genesis/prod/02601.ofx"
    query.Signon.Org = ofxgo.String("ISC")
    query.Signon.Fid = ofxgo.String("2601")
    

    I keep getting no bank messages received using the example. with my credentials and bank id and acctid in place.

Go library containing a collection of financial functions for time value of money (annuities), cash flow, interest rate conversions, bonds and depreciation calculations.

go-finance Go library containing a collection of financial functions for time value of money (annuities), cash flow, interest rate conversions, bonds

Jan 2, 2023
Payment abstraction library - one interface for multiple payment processors ( inspired by Ruby's ActiveMerchant )

Sleet Payment abstraction library - interact with different Payment Service Providers (PsP) with one unified interface. Installation go get github.com

Dec 28, 2022
Simple and easy to use client for stock market, forex and crypto data from finnhub.io written in Go. Access real-time financial market data from 60+ stock exchanges, 10 forex brokers, and 15+ crypto exchanges

go-finnhub Simple and easy to use client for stock, forex and crpyto data from finnhub.io written in Go. Access real-time market data from 60+ stock e

Dec 28, 2022
money and currency formatting for golang

accounting - money and currency formatting for golang accounting is a library for money and currency formatting. (inspired by accounting.js) Quick Sta

Dec 21, 2022
Matching Engine for Limit Order Book in Golang

Go orderbook Improved matching engine written in Go (Golang) Features Standard price-time priority Supports both market and limit orders Supports orde

Dec 22, 2022
Package to easily consume the Dolarpy API in golang.

dolarpy-go Package to easily consume the Dolarpy API in golang. https://github.com/melizeche/dolarPy - by melizeche Install import "github.com/bitebai

Apr 11, 2022
Go-finproto - a collection of finance-related protocols implemented in Golang

go-finproto go-finproto is a collection of finance-related protocols implemented

Dec 25, 2022
A go port of numpy-financial functions and more.

go-financial This package is a go native port of the numpy-financial package with some additional helper functions. The functions in this package are

Dec 31, 2022
This is a backend of wallet app for personal spending and income management.

wallet-ap-graphql this is a backend of wallet app for personal spending and income management. technologies used: golang, graphql, postgres, redis, jw

Jan 12, 2022
Golang library for querying and parsing OFX

OFXGo OFXGo is a library for querying OFX servers and/or parsing the responses. It also provides an example command-line client to demonstrate the use

Jan 3, 2023
Kick dropper is a very simple and leightweight demonstration of SQL querying, and injection by parsing URl's

__ __ __ __ _____ ______ | |/ |__|.----.| |--.______| \.----.| |.-----.-----.-----.----.

Feb 6, 2022
ByNom is a Go package for parsing byte sequences, suitable for parsing text and binary data

ByNom is a Go package for parsing byte sequences. Its goal is to provide tools to build safe byte parsers without compromising the speed or memo

May 5, 2021
Go library for querying Source servers

go-steam go-steam is a Go library for querying Source servers. Requirements Go 1.1 or above Installation go get -u github.com/sostronk/go-steam To us

Oct 28, 2022
Example of querying the balance of Crypton and UUSD with Utopia Ecosystem API and utopialib-go

account-balance-go Example of querying the balance of Crypton and UUSD with Utopia Ecosystem API and utopialib-go example of use flags: -host string

Oct 8, 2021
Golang client for querying SecurityTrails API data
Golang client for querying SecurityTrails API data

haktrails haktrails is a Golang client for querying SecurityTrails API data, sponsored by SecurityTrails. SecurityTrails $50 Bug Bounty Hunter Plan Sa

Jan 4, 2023
Golang Client for querying Tor network data using the Onionoo service.

gonion Lightweight Golang wrapper for querying Tor network data using the Onionoo service. package main import ( "github.com/R4yGM/gonion"

May 11, 2022
Automatically capture all potentially useful information about each executed command (as well as its output) and get powerful querying mechanism
Automatically capture all potentially useful information about each executed command (as well as its output) and get powerful querying mechanism

nhi is a revolutionary tool which automatically captures all potentially useful information about each executed command and everything around, and delivers powerful querying mechanism.

Nov 29, 2022
Build a local copy of Known Exploited Vulnerabilities Catalog by CISA. Server mode for easy querying.

go-kev go-kev build a local copy of Known Exploited Vulnerabilities Catalog by CISA. Usage $ go-kev help Go Known Exploited Vulnerabilities Usage:

Oct 30, 2022
Build a local copy of Known Exploited Vulnerabilities Catalog by CISA. Server mode for easy querying.

go-kev go-kev build a local copy of Known Exploited Vulnerabilities Catalog by CISA. Usage $ go-kev help Go Known Exploited Vulnerabilities Usage:

Oct 30, 2022
Command line utility for querying AWS Athena, seeks to emulate sqlite3

athena-query Command line utility for querying AWS Athena, seeks to emulate sqlite3. It implements some similar commands (the . notation) as sqlite3.

Jan 30, 2022