You wanted it? Here it is.
Recent Weblog Entries
Developer Stats
Height: 5'9"
Favorite Drink: Mountain Dew
Hobby: Coding
Favorite Languages: PHP and JS
Archive By Category
- Web Development (2)
- Personal (0)
- Site Changes (0)
- Productivity (0)
- Thoughts (1)
- Code (0)
Creating a Mint.com API
I have recently started using Mint.com for my personal finances. At first, I was very skeptical about the security of things at mint. I was unsure whether or not I could trust the connection and my personal banking information to be accessed by another person and/or company. In short, after reviewing their security policy and taking a closer look, I was able to move past my insecurities.
With ease, I have been able to connect all of my banking information in just a few simple clicks… Pretty scary, huh? I think this is an amazing feature for banking 2.0 sites. Mint.com takes care of organizing your finances through an easy-to-use interface. All this is great but, as a developer, my mind started thinking about how I can make use of this site. And so, the task of investigating whether or not mint.com has an official API to pull transactions, account balances, and other things of the like. It would be great if I could integrate mint.com support into different applications and backend’s that I write.
After a couple hours of perusing through the Internet’s web of data, I was able to find a large amount of user requests for exactly this sort of thing. Seems like I’m not the only one to have had this idea, nothing new here… While there are definitely pro’s and con’s of having an API to access account information, I think that this feature would prove very beneficial to the site. There are a multitude of things I could use mint.com API support for. Anyways, without deviating from my original point… Wait, what’s that? … You guessed it! I began to play with the idea of creating a high-level PHP class to support scraping mint.com for user information.
Within around an hour or so, I was able to automate the mint.com login and scrape all of my accounts, including:
- Account Name
- Time last checked
- Account Balance
- Account Nickname
It was actually rather easy to do. It’s a simple case of using PHP and cURL to fetch the information you need. It looks a little something like this:
$url = 'https://wwws.mint.com/loginUserSubmit.xevent'; $post = array( 'username' => $username, 'password' => $password, 'task' => 'L' ); //build the query _GET string $post = http_build_query($post); $ch = curl_init($url); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_TIMEOUT, 12); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $post); curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt'); //get the results now $result = curl_exec($ch); $error = curl_error($ch); curl_close($ch);
That bit of code sets up a cURL object that contains all of the appropriate settings/POST information that it needs to log into the mint.com site. This cURL request requires a cookie jar to store the result of the mint.com login script(s). After you have the result, you can actually begin scraping out the individual accounts by exploding on the text “</a></h4><h6><span>” and filtering through each match to make sure that you can pull the information that you want.
Now that we’ve generally covered the scraping process, you can take it another level by performing things like:
- Retrieving account transactions
- Performing a refresh account request
- Getting account alerts
- Pulling budget information
- Getting mint.com charts
- And all of the other great stuff Mint.com has to offer…
In the hour I had to experiment with this idea, I was able to toy with the idea of performing a “Refresh Account” request. This request, along with the others i just mentioned in the list above, require a little more work. I still have not perfected the request due to lack of time. But from what it looks like, mint.com initiates a server-side request to refresh all accounts attached to the current user. A token is passed to the refresh script as another layer of security. After reviewing the source of the overview page, it looks like the token can be directly pulled from the html, so you would need to simply extrapolate this data in order to perform tasks. I have tested this by just posting the extracted token to the url, along with the original cookie jar, and I receive a response: “Session has expired.” This makes me believe that mint.com is expecting information from cookies that have been generated client-side. The next step is to basically pull the data that is generated client-side and post it to the url above, along with the token and the cookie jar. The token can be found in a hidden input with the id “javascript-user”. I think the naming convention confirms my thought process about client-side cookie generation. I will need more time to look into this further and have plans to post my findings when I get to them.
The reason the refresh is so important is because of the slow HTTPS connection that occurs when you log in with cURL and PHP. To speed this up, we could auto-cache a first result of the accounts and their balances, much like what mint.com does. After the accounts have been cached, we can pull the data from MySQL almost instantly and then give an “Update All” button to the user just like mint.com. This would help us achieve faster page load speeds as we won’t be waiting for a 10 second script to finish loading dynamically.
I say all of this to mention that getting mint.com account data isn’t all that bad… However, I’d prefer an official REST API or something of this sort.
I am only trying to get at my transactions, which should be accessible via this URL once I am authenticated:
https://wwws.mint.com/transactionDownload.event?accountId=5077467 (for each account)
I am using PHP 5.2.6 from Debian Lenny, with Curl 7.18.2
1) the Mechanize library
2) beautifulSoup
those two libraries are great for manipulating data via scripting like Ruby or Python...
Also, there is a CSV export option in Mint. Could this server-side login-method be used to create an always-available CSV source link so that we can constantly pull it into apps that support web connect?
From there, you could put it into a Yahoo! Pipe and manipulate the data even further, with output formats like JSON, XML, or RSS
At this point, you'd have a PHP array full of all the data you need. You could loop through and build a CSV file whenever you want. I'd just be interested in initiating some sort of refresh call to their site via a script. When I get some more time, I'm going to look into this a little more.
//retrieves CSV file of transactions function mint_get_transactions_csv($foreign_account_id=false) { //set the parameters $url = 'https://wwws.mint.com/transactionDownload.event'; if ($foreign_account_id) $url .= '?accountId=' . $foreign_account_id; //build the request $ch = curl_init($url); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_TIMEOUT, 12); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/curl_cookies.txt'); curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/curl_cookies.txt'); //get the response $response = curl_exec($ch); $error = curl_error($ch); curl_close($ch); if ($response) return $response; else return false; }Turning the CSV into an array is easy, just explode it a couple of times:
if ($csv) { $transactions = array(); $csv_lines = explode("\n", $csv); //convert file into an array of lines unset($csv_lines[0]); //remove the header line foreach ($csv_lines as $line) { //munge the incoming data into some semblance of order $line = substr($line, 1, strlen($line)-2); //strip off first and last quotes $line = explode('","', $line); //convert line into an array of fields /* .... your custom stuff here .... */ }Beware that Mint returns one line of the file that doesn't explode properly - it's probably just a trailing hard return at the end. I'm ignoring several of Mint's fields so further code wouldn't be generally useful, but the next step would be converting the array into an associative array for easy use. There's probably a nifty way to do this with the contents of that first line that I simply delete. It's too bad that Mint doesn't provide unique ID's, because that is the only way to discern between altered lines and new lines, for certain.
I am putting the transactions into my own database at this point, so Mint is simply providing me with a highway to all of my financial institutions' data. Next I am going to push new transactions to an RSS feed (accessible in Google Reader and from my phone) where I will see and categorize them, and from there into a Google spreadsheet for consumption in whatever form I care to create. Win! One of the most important reasons was to be able to use my mere 12 categories instead of Mint's 80+, which cannot be made convenient on a phone platform by any UI trick known to man. Why Mint doesn't allow hiding categories itself, doesn't push to RSS/phone for categorization, doesn't mark reviewed vs. un-reviewed transactions, and doesn't allow historical/manual transaction insertion remains a mystery. But now I don't have to care.
I'd try again but I'm not sure what went wrong and don't want to spam you.
Here's the mobile-friendly page I'm using for handling unreviewed transactions: http://www.ethanpooley.net/bnfo.jpg
Easy to put in a desktop web-widget also.
What do you think? Let me know.
* Bolded fields are required.